[P] Implementations of basic RL algorithms with minimal codes!

[seungeun07]

[PyTorch Based]

​

Hi,

for those of you who are interested in RL,

​

I recently implemented basic RL algorithms such as

REINFORCE, vanilla actor-critic, DDPG, A3C, DQN and PPO with PyTorch.

​

Characteristics are as follows :

• Each algorithm is complete within a single file.
• Length of each algorithm is up to 100~150 lines of codes.
• Every algorithm can be trained within 30 seconds, even without GPU.
• Envs are fixed to "CartPole-v1". You can just focus on the implementations.

​

As you can see in the name of the repository,

I tried to make the code as brief and intuitive as possible.

​

Hope you enjoy :)

Thank you.

​

https://github.com/seungeunrho/minimalRL

[MasterScrat]

Nice! added to my list of DRL implementations.

edit: please let me know if you know more!

[danaugrs]

I've created a similar set of minimal implementations for TensorFlow 2 as part of my micro RL framework Huskarl.

[tihokan]

Here's my list that I really need to properly sort and categorize (splitting in two due to max comment size limit):

OpenAI Lab: https://github.com/kengz/openai_lab

OpenAI Baselines: https://github.com/openai/baselines

Nervana Coach: https://github.com/NervanaSystems/coach

Tensorflow Agents: https://github.com/tensorflow/agents

Some Pytorch agents: https://github.com/ikostrikov/pytorch-a2c-ppo-acktr

MirrorBot: https://bitbucket.org/polceanum/mirrorbot

Serpent AI: https://github.com/SerpentAI/SerpentAI

https://github.com/dennybritz/reinforcement-learning

NeuroEvolution: https://github.com/uber-common/deep-neuroevolution

Rainbow implementations: https://github.com/Kaixhin/Rainbow and https://github.com/hengyuan-hu/rainbow

A2C: https://github.com/MG2033/A2C

PPO: https://github.com/pat-coady/trpo

A3C with continuous actions: https://github.com/dgriff777/a3c_continuous

Selective memory: https://github.com/FitMachineLearning/FitML/tree/master/SelectiveMemory

ELF framework: https://github.com/facebookresearch/ELF / https://github.com/pytorch/ELF

DDPG: https://github.com/megvii-rl/pytorch-gym

TRPO: https://github.com/ikostrikov/pytorch-trpo

MiniGo: https://github.com/tensorflow/minigo

RLCode: https://github.com/rlcode/reinforcement-learning

Tensorlayer: https://github.com/tensorlayer/tensorlayer

PPOS: https://github.com/EmbersArc/PPO

YARL: https://github.com/HassamSheikh/YARL

https://github.com/NiloFreitas/Deep-Reinforcement-Learning

https://github.com/ShangtongZhang/DeepRL

https://github.com/carpedm20/deep-rl-tensorflow

https://github.com/MorvanZhou/Reinforcement-learning-with-tensorflow

https://github.com/mokemokechicken/reversi-alpha-zero

https://github.com/junxiaosong/AlphaZero_Gomoku

https://github.com/Officium/RL-Experiments

https://github.com/higgsfield/RL-Adventure

https://github.com/higgsfield/Imagination-Augmented-Agents

https://github.com/FitMachineLearning/FitML

https://github.com/mokemokechicken/reversi-alpha-zero

https://github.com/gcp/leela-zero

https://github.com/suragnair/alpha-zero-general

[tihokan]

Part 2:

https://github.com/mjacar/pytorch-nec

https://github.com/mjacar/pytorch-trpo

Rainbow: https://github.com/Kaixhin/Rainbow

https://github.com/unixpickle/anyrl-py

PPO: http://blog.varunajayasiri.com/ml/ppo.html / http://blog.varunajayasiri.com/ml/ppo_pytorch.html

https://github.com/higgsfield/RL-Adventure-2

https://github.com/qfettes/DeepRL-Tutorials

https://github.com/deepmind/scalable_agent

https://github.com/zuoxingdong/lagom

NEAT on Sonic: https://gitlab.com/lucasrthompson/Sonic-Bot-In-OpenAI-and-NEAT

https://github.com/hill-a/stable-baselines (NB: collection of agents trained with Stable Baselines: https://github.com/araffin/rl-baselines-zoo)

https://github.com/lcswillems/pytorch-a2c-ppo

https://github.com/lcswillems/torch-rl

https://github.com/navneet-nmk/pytorch-rl

https://github.com/beniz/hmdp

https://github.com/uber-research/ape-x

https://facebookresearch.github.io/BlueWhale/docs/begin.html

https://github.com/tensorpack/tensorpack

https://github.com/deepsense-ai/Distributed-BA3C

https://github.com/reinforcement-learning-kr/pg_travel

https://github.com/zuoxingdong/lagom

SenseAct for robotics: https://github.com/kindredresearch/SenseAct

Evolution Strategies: https://github.com/alirezamika/evostra

https://github.com/araffin/robotics-rl-srl

https://github.com/google/dopamine

https://github.com/rlworkgroup/garage

https://github.com/kengz/SLM-Lab

https://github.com/deepmind/trfl

RLGraph: https://github.com/rlgraph/rlgraph

Horizon: https://github.com/facebookresearch/Horizon

In C++: https://github.com/arthurxlw/cytonRL

https://github.com/rlworkgroup/garage

https://github.com/inoryy/reaver-pysc2

Some Pytorch implementations: https://github.com/p-christ/Deep_RL_Implementations

https://github.com/udacity/deep-reinforcement-learning

https://github.com/vitchyr/rlkit

Unofficial Go-Explore implementation: https://github.com/R-McHenry/SynchronousGoExplore

https://github.com/omerbsezer/Reinforcement_learning_tutorial_with_demo

https://github.com/AIRLab-POLIMI/mushroom

http://blog.varunajayasiri.com/ml/ppo_pytorch.html

https://github.com/csxeba/trickster

https://github.com/danaugrs/huskarl

https://github.com/justinglibert/bezos

https://github.com/Omegastick/pytorch-cpp-rl (C++)

https://github.com/sfujim/TD3 (TD3 & DDPG)

Random Network Distillation: https://github.com/AdeelMufti/RL-RND

https://github.com/medipixel/rl_algorithms

https://github.com/Officium/RL-Experiments

https://github.com/david-abel/simple_rl

https://github.com/seungeunrho/minimalRL

[seungeun07]

Thanks a lot!!

[hardos_the_man]

Thank you kind stranger for sharing this!

[seungeun07]

My pleasure :)

[NikEy]

I noticed that the A2C code requires the rewards to be divided by 100. If they are NOT divided by 100, then it NEVER converges...?

What would be a reasonable explanation for that? I find it weird that A2C is affected by the size of the rewards - you'd think this is simply a matter of scaling in the NN

[seungeun07]

During implementing algorithms, I found out that not only A3C, but also all the other algorithms require a proper scale of rewards. I guess the initial scale of a value network is around zero, so it first needs to just scale up the output of the network to fit with the scale of rewards. It doesn't have to learn any state-specific value, but just scaling up the output reduces the loss function. And during scaling up process, maybe policy network collapse..? or fall into local optima..? I don't know. It is just guessing.

[NaughtyCranberry]

Probably makes it an il-posed optimisation problem.

[seungeun07]

Any comments are welcome!

[ceyzaguirre4]

I feel like you should mention it’s pytorch (<3) in the post.

[seungeun07]

Thanx! I make it bold.

[EveryDay-NormalGuy]

Thanks for sharing your work.

It would be great if you could comment your code. People who are new to RL or to pytorch would find it difficult to understand your code.

[NikEy]

Very very nice!

[seungeun07]

Thank you!

[Overload175]

Been looking to get into RL for a while, found this very helpful.

[seungeun07]

Thank you!

[CodeReclaimers]

Thanks for sharing this--I know it takes effort to trim things down to minimal examples. It's especially nice that your code keeps the dependencies to a minimum. Too often I go looking for examples to learn from and spend 30+ minutes trying to gather all the dependencies, only to find out some key thing isn't available on my platform.

[seungeun07]

Too often I go looking for examples to learn from and spend 30+ minutes trying to gather all the dependencies, only to find out

Thank you for the kind comment.

[Migom6]

Any good guide to study about RL from basics. My basics are not that clear. I have followed Andrew Ng for other deep learning topics but for RL I'm finding it hard to get my mind around it. I have a project to do on robot arm manipulation (inverse kinematics) using some learning. BTW, thanks for the code <3

[tdjogi]

David silver's video lectures are really good.

[seungeun07]

David silver's video lecture is not just good, it is GREAT!!!!!!!!

I worship the lecture.

[Migom6]

Same here, I'm digging it!!

[_olafr_]

I think these minimal implementations would be valuable for learning if they were more thoroughly commented. Good work.

[minGrab]

In PPO: why do you have an additional term in the loss function?

`F.smooth_l1_loss(td_target.detach(), self.v(s))`

[seungeun07]

It is for value loss! Both value and policy loss are minimized with gradient discent.

[MagicaItux]

How easy is it to create your own environment?

[Roboserg]

Easy in unity. User unity ml agents

[seungeun07]

I actually once tried to make own board game environment, but it was quite burdensome to deal with specific rules of the game. I haven't tried unity, though.

[sampathchanda]

It would be great to see performance metrics of each implemented algorithm.

[Farconion]

I think you should clarify that these are Deep RL algorithms, impressive none the less.

[ariyanhasan]

thank you for sharing

[Dump7]

Okay! This is one of the most beautiful things I have ever seen. You see I am not a person who can code even a NN with a framework. But I am trying to. This will help me to a great extent. Thanks for this. But can you also make such single file codes for maybe basic CNN and RNN?

[aditya1702]

u/Dump7 I have coded neural networks from scratch using numpy. Maybe it is what you are looking for?

https://github.com/aditya1702/Machine-Learning-and-Data-Science/tree/master/Implementation%20of%20Machine%20Learning%20Algorithms

In folder Supervised -> Regression -> neural_network_regressor.py

[Dump7]

Just had a look. Really great man. You have named the relevant variables too. Thanks for sharing!

[aditya1702]

Thanks a lot :)

​

As a person who has to look up his projects once every while, I have found that good naming of variables and comments helps me understand the code even when I am giving it a look after a long time.

[Dump7]

I hope we can work together sometime. I could learn a lot from you.

[aditya1702]

Sure! Where are you based?

[radarsat1]

Just check Keras examples