The future of Gym & conflict of interest

[tristandeleu]

This is not a technical issue, but an issue/discussion about the recent changes regarding the maintenance of Gym. I am opening this issue here since there is no "Discussion" section in this repository yet.

Over the past few days, I've seen a clear trend where SuperSuit and PettingZoo have been mentioned and pushed forward a lot by the new maintainer of Gym @jkterry1 and other contributors from the Petting-Zoo project. This includes personal opinions stated as facts (e.g. "SuperSuit [...] is better for this by every measure"), and censoring messages (e.g. a message by @semitable was removed for expressing a -reasonable- opinion about PettingZoo).

This looks a lot like conflict of interest in my opinion: these two libraries in particular are being pushed forward specifically because the new maintainer(s) are working on them. I understand that the experience acquired working on these libraries can be translated to Gym to some extent, but this is going beyond that. Promoting personal libraries this much is not what should be expected of Gym maintainers.

My suggestion moving forward is for the Gym maintainers to stop explicitly promoting their personal libraries (e.g. remove them from the README), and to limit mentioning them to the minimum; ideally if there is something that is worth taking from the Petting-Zoo projects, then this should come from the Gym community, and not be pressured by the current maintainers (e.g. adding SuperSuit features).

As for the future of Gym, I would like to echo this comment: while it is exciting to see new maintenance of Gym, I strongly encourage taking things slow, and not try to change too much too quickly. I encourage the new maintainers to go through the code in order to get a better understanding of the codebase first before making any further changes. This would avoid comments such as "I managed to become the maintainer of Gym without knowing [these wrappers] even existed", or misconceptions, and it will help building the documentation.

As one of the top contributors to Gym, I also invite the maintainers to listen to the community: if there is no Github issue/PR regarding a specific feature, then making deep changes, or even breaking changes should probably not be the priority.

[jkterry1]

-So again, I pulled SuperSuit from Gym per community feedback, and I apologize for committing that directly to master. I have no problem removing it from the readme given the path we're taking and just did so.
-The issue with removing the comment was discussed to death in the #practical channel of the RL discord if you'd like to see the resolution of that in more detail, but you are correct that I should not of done that and I publicly apologize.
-We can discuss this further if you'd like, but I would argue that PettingZoo is sufficiently relevant to link in the readme. If something analogous to Gym with widespread use arrises in other domains (e.g. offline RL), I would sincerely like to link to such works in the readme as well. Beyond linking from the readme and discussing porting relevant features over (e.g. in #2289), I don't believe I've unnecessary promoted PettingZoo in particular.
-The specific features I wish to add to Gym related to SuperSuit are common things that Gym is missing because the preprocessing wrappers haven't seen a lot of focus in the past years. SuperSuit, unsurprisingly, includes my attempt at a comprehensive list of the common and useful RL wrappers. With a small exception they are not breaking. I also do not see how including features constitutes promoting SuperSuit. It constitutes adding things that are useful, and ties into issues regarding new changes.
-I appreciate your encouragement to review the Gym codebase :). My point in that comment fairly clearly was just that those three wrappers I discussed were, to the best of my knowledge, not exactly widely used compared to others.

My goal in maintaining Gym, as I reiterated to Antonin Raffin in that thread, is to not break things upstream with the exception of vector environments. I agree in the importance of this. I have good reason for wanting to break vector environments in that the maintainers of many open source RL libraries (CleanRL, the Autonomous Learning Library, the current maintainer of the Arcade Learning Environment, and for instance SB3 created their own vector environment structure) have all expressed serious misgivings with the API, many of which are reiterated in that thread (though the maintainer of the ALE has yet to comment as he's working on a deadline), and even OpenAI themselves doesn't really use it anymore due to technical issues with it (they use Gym3). A specific thing that I would like to personally address with vector environments is that, for detailed technical reasons, using Gym is highly discouraged at places like Google Brain, for reasons the maintainer of the ALE will elaborate on himself soon. If virtually everyone doesn't want to use your API than, in my opinion, breaking it such that people will use it is the prudent course of actions. You may disagree with this philosophy, but this is mine. Beyond this I do recognize the importance of not breaking Gym and am on board with you there.

Your final point, and please clarify if I'm misunderstanding, appears to be that no new features should be added unless a community member creates an issue requesting it on the Gym repo? I very strongly disagree. For one, people didn't create new issues here for ages. Also, sometimes people just don't create issues for things. In one notable instance for example, OpenAI had planned to transition from Atari-Py to ALE-Py for something like 2 years to solve a host of performance and maintenance issues with Atari environments. I intend to continue with this plan for a variety of reasons (there are /huge/ maintenance issues with this), but it will result in a nonzero number of breaking changes. To my knowledge, no one has ever created a Github issue for this, even the huge maintenance problems that motivated the shift. You may argue that I should create GitHub issues for every large planned changes like this, which isn't an unreasonable request, but I am one man and am limited by time and energy.

To recap:

• I pulled SuperSuit from Gym and the readme and I disagree that anything currently going on could reasonably represent a conflict of interest
• I think you were primarily referring to SuperSuit, but if you weren't I do not track how my actions regarding PettingZoo are unreasonable.
• I disagree that all new changes need to come from proposals in organically created GitHub issues or similar
• I strongly believe that we should not break things in Gym where possible, as I keep saying. I am of the belief that it is uniquely worth it for vector environments. I will seriously consider all your opinions, and those of others, in the vector environment thread, but at a certain point professional technical disagreements may exist with any project.

[tristandeleu]

My concerns regarding conflicts of interest are for both SuperSuit and PettingZoo. The comment that was deleted was about PettingZoo specifically, and I can't remember another instance when a comment had been deleted in the past. This happened to be about your personal library, so it is clearly conflicting with your new position as a maintainer of Gym.

I also do not see how including features constitutes promoting SuperSuit.

On the specific case of SuperSuit, I think it would be best to see if the community wants these features in the first place (through feature requests/issues), rather than pushing for these features yourself. The latter does constitute conflict of interest. And I hear your concern regarding only looking at Github issues/PRs for new features, but specifically for wrappers, almost all of those available in Gym came from community requests.

I have good reason for wanting to break vector environments in that the maintainers of many open source RL libraries (CleanRL, the Autonomous Learning Library, the current maintainer of the Arcade Learning Environment, and for instance SB3 created their own vector environment structure) have all expressed serious misgivings with the API, many of which are reiterated in that thread (though the maintainer of the ALE has yet to comment as he's working on a deadline), and even OpenAI themselves doesn't really use it anymore due to technical issues with it (they use Gym3). A specific thing that I would like to personally address with vector environments is that, for detailed technical reasons, using Gym is highly discouraged at places like Google Brain, for reasons the maintainer of the ALE will elaborate on himself soon. If virtually everyone doesn't want to use your API than, in my opinion, breaking it such that people will use it is the prudent course of actions. You may disagree with this philosophy, but this is mine.

I don't disagree with this philosophy. This discussion is best left in #2279, but if you can detail these good reasons there that will be very helpful.

Your final point, and please clarify if I'm misunderstanding, appears to be that no new features should be added unless a community member creates an issue requesting it on the Gym repo?

What I suggest (and echoing @araffin's comment) is going slow in terms of new features, at least at the beginning, and add/fix what comes up through the issues/PRs first. That's what you've been doing so far and that's great, and I suggest continuing this way at least until you get every non-user-facing-code changes done (documentation, fixing typos, tests). This is what I meant by "not the priority".

[jkterry1]

-Regarding PettingZoo, I deleted the comment because the reply to the first half was "dude read the thread" and the reply to the second half is "the data and using the library would generally suggest otherwise" and I was tired. People delete comments with people because they're annoying them all the time. Without dragging others into drama, this has happened to me on other major RL libraries before.
-Regarding proposing the inclusion of features SuperSuit has, I'm not going to argue with you on this but like that literally isn't what "conflict of interest" means if you Google it. I also think that the reason wrappers are incomplete is that they've only come from community suggestion.
-Regarding your last point, I appreciate your input on how I should fulfill my role as the maintainer of Gym and I will take that into consideration going forward. Ultimately though, as this is unpaid volunteer work, as long as I'm not breaking things others depend on or am somehow profoundly mismaintainting Gym, what I enjoy spending my time working on is up to me?

As an aside because you keep mentioning it, dude people are extensively working on documentation and a website and so on. It's going to take awhile, probably till the end of the month. Other things will happen between now and then, and that's okay.

[jkterry1]

One other thing that I would like to emphasize, is that while I appreciate your input here, you are not my boss and you are not the ultimate arbiter of what I should be doing with Gym and many of your comments, at least to me personally, strongly come off with that tone. I value your suggestions on technical matters with regard to Gym, but I would respectfully request that you keep this in mind.

[tristandeleu]

I have to say, this reaction is unfortunate.

[jkterry1]

I'd like to clarify a few points after someone read this and messaged me:

• I would like to clarify that I understand that I am not the sole arbiter of what happens with Gym going forward either, and I accidentally implied that. I very much believe community feedback is needed on all future changes.
• To again reiterate, my plan going forward is to not break important features in Gym or the large ecosystem of libraries that depend on it.
• I want to be very clear that I am not trying to do a hostile takeover of Gym to somehow support PettingZoo and SuperSuit. My suggestions for Gym involving SuperSuit were motivated by my belief that they would be the best path forward to solve problems with Gym. I believe that I have responded to community feedback adequately by reverting those changes and amending my plans--after reverting those changes the only mention of PettinZoo or SuperSuit in gym is 1 sentence in the README, and there are no plans to involve or depend on PettingZoo or SuperSuit in the future, besides borrowing some technical ideas. I will also stand by my promise of linking to APIs for other domains (e.g offline RL) that achieve nontrivial levels of standardization in the future.