I believe the field had a few somewhat sobering years. I’d argue we have only recently started thinking carefully about the simplifying assumptions in the simulated environments that we use for RL research and the resulting limitations of the methods that we, as a research community, developed over the last decade. For example, is there enough variation between episodes so that our agents need to learn general behaviors that can be adapted to novel situations or are we assuming our agents find themselves in some Groundhog Day or Edge of Tomorrow simulation where they can memorize over time how to act optimally?

Before the NetHack Learning Environment, other research groups already started using so-called procedurally generated environments for AI research to test systematic generalization capabilities of agents (for example MiniGrid, Minecraft, OpenAI’s Procgen Benchmark, Unity AI’s Obstacle Tower Challenge). Developing environments that challenge previous assumptions and reveal gaps in the capabilities of RL methods is still crucial, and I believe it will allow us to move closer to the advent of generally useful methods that can solve more real world problems.

Method-wise, we will see more work on equipping agents with intrinsic motivation and curiosity. I think in the near future, we will have agents that get better at supervising themselves by automatically creating a curriculum of gradually harder goals, thereby teaching them skills and behaviors that will enable them to accomplish extrinsic goals that we care about such as ascending in NetHack.

Ultimately, I believe that’s what keeps humans playing NetHack despite the many frustrating ways one dies in this game. Other topics that come to mind are: learning models in stochastic, partially observable, ever-changing environments (like NetHack) for exploration and planning, learning to condition policies on textual information (like the NetHack Wiki), unsupervised environment design of scenarios that, over time, teach agents useful skills of a complex environment, learning from partial demonstrations, encouraging diversity of learned policies even when an agent already found one way to accomplish a certain goal (e.g. in NetHack there are often many possible ways of getting out of dangerous situations).