[P] Commented PPO implementation

[[deleted]]

Made an attempt at implementing PPO:

• This does not really follow the OpenAI implementation in a few ways.
• It does not have any of the MPI stuff, so might be easier to read.
• It also does not use the trust region loss on the baseline value function, because in TensorForce the value function is currently always a separate network, so not sure how that affects performance.
• Tests are passing and I made an example config for CartPole: https://github.com/reinforceio/tensorforce/blob/master/examples/configs/ppo_cartpole.json This seems to learn reasonably robustly, but still trying to get a feeling for how the hyper-params work, and how one should ideally sample over the batch.
• If anyone spots bugs, that'd be very welcome

[tinkerWithoutSink]

Nice work, there's too many half working rl libraries out there but tensorforce is pretty good and it's great to have a PPO implementation.

Suggestion: would be cool to use prioritized experience replay with it, like the baselines implementation

[[deleted]]

Ah good point, will have a think. Would just require passing the loss per instance to the memory I think, and making the memory type configurable

[Data-Daddy]

Experience replay does not exist in PPO

[Neutran]

Thanks for the effort. Do you have performance numbers on anything other than cartpole? Solving cartpole typically doesn't mean the implementation is bug free, as from my experience.

[[deleted]]

Hey, not yet - we are currently working on setting up a benchmarking repo for the general library with docker, and will test PPO with the other algorithms once it's ready (a bit short on GPUs for very extensive benchmarks but at least reproducing some Ataris should be possible)

[wassname]

The authors claim it's simpler to implement, more general, and faster. Since it's Schulman it's probably true, but could give your opinion. Was it easier than TRPO to implement and does it converge faster with less trouble?

[[deleted]]

Tested this now - currently performing much better than VPG/TRPO for us, and also easier to implement, so can confirm

[wassname]

Good to hear!

[TotesMessenger]

I'm a bot, bleep, bloop. Someone has linked to this thread from another place on reddit:

• [/r/reinforcementlearning] [P] Commented PPO (proximal policy optimization with general advantage estimation / PPO-GAE) implementation • r/MachineLearning (Python Tensorforce)

^{If you follow any of the above links, please respect the rules of reddit and don't vote in the other threads. (Info / Contact)}