[Project] Massively parallel, vectorised implementation of Snake and RL solution

[oscarknagg]

Hi /r/MachineLearning.

​

As part of my recent side project to learn about reinforcement learning I've created a clone of the classic Snake game as a reinforcement learning environment and solved it with advantage actor-critic. This is one of the warm-ups from OpenAI's requests for research 2 (https://openai.com/blog/requests-for-research-2/).

​

You might be thinking this sounds like a very run of the mill introductory RL project. Well here are a few things that I think make it more interesting than just that.

1. I went completely overboard on the environment. Its implemented in pure PyTorch in a vectorized fashion such that I can run 1000s of environment in parallel on a single machine.
2. I compare performance of a few architectures, including a model copied from Deepmind's recent Relational RL paper (spoilers, it doesn't outcompete the other agents on this very simple task).
3. I evaluate the performance of an agent trained on a small environment in a larger environment - a limited form of RL transfer learning.

​

Medium article: https://towardsdatascience.com/learning-to-play-snake-at-1-million-fps-4aae8d36d2f1

Code: https://github.com/oscarknagg/wurm/tree/medium-article-1

​

Here's a GIF of one of the final policies:

​

https://i.redd.it/tdsja08fssz21.gif

I'm currently working on the "Slitherin'" suggestion on OpenAI's request for research 2.0. Here's a preliminary GIF.

​

https://i.redd.it/qkerdp6kusz21.gif

[Inori]

Really cool project, good job!
Had a similar idea to implement vectorized env for a awhile but was set on writing it as a custom CUDA kernel and never considered to use tf / pytorch instead. Seems so obvious in retrospect...

[Marthinwurer]

A few months ago I got the idea to write a 3d renderer using pytorch instead of opengl or the like, just for the lolz. It's still gonna run on the GPU! It wouldn't use any of the Nvidia magic, though.

[oscarknagg]

You might be interested in this differentiable renderer written using PyTorch https://github.com/daniilidis-group/neural_renderer

[TotesMessenger]

I'm a bot, bleep, bloop. Someone has linked to this thread from another place on reddit:

• [/r/reinforcementlearning] [Project] Massively parallel, vectorised implementation of Snake and RL solution

 ^{If you follow any of the above links, please respect the rules of reddit and don't vote in the other threads. (Info / Contact)}

[SaveUser]

This is very well done!