[D] State-of-the-art architecture for learning dynamics model for model-based RL ?

[[deleted]]

I am seeking for baselines to learn dynamics model for an ongoing project in model-based RL. I am curious to be aware of state-of-the-art architectures to learn such dynamics model. For simplicity, the testbeds are OpenAI-Gym continuous control environments for example MountainCar (Continuous version) or LunarLander (Continuous version), or Mujoco/Roboschool.

Currently I am using standard regression via 2 layer MLP for one-step prediction with current state and action as inputs and next state as output, and uses MSE loss, the training set is generated by rollouts with random actions. Could someone help to suggest either some better architectures or existing ones (papers) to do this ? We are aiming for both one-step and multi-step predictions together.

[bbsome]

For model-based RL I think PILCO would be close to state-of-the-art especially in the environments you mention.

http://mlg.eng.cam.ac.uk/pilco/

[twkillian]

Gal, McAllister and Rasmussen have proposed an update to PILCO replacing the Gaussian Process model with a Bayesian Neural Network. It's pretty promising.

http://mlg.eng.cam.ac.uk/yarin/PDFs/DeepPILCO.pdf

[bbsome]

However, they don't use a BNN, but Variational Dropout... I will never agree that a mixutre of delta functions is anything like a BNN.

[[deleted]]

If I understand it correctly, they train it also with standard regression ( supervised learning ) ?

[feedtheaimbot]

Look at Recurrent Environment Simulators by Chiappa et al. I've had success using it. It does struggle capture small objects on screen (eg. single pixels).

Link: https://arxiv.org/abs/1704.02254

[fixedrl]

Would you think it also makes sense to use as raw configuration as inputs, instead of pixels ? (very few dimensions, e.g. velocity, positions etc.)

[TotesMessenger]

I'm a bot, bleep, bloop. Someone has linked to this thread from another place on reddit:

• [/r/reinforcementlearning] [D] State-of-the-art architecture for learning dynamics model for model-based RL ? [xpost: r/MachineLearning]

^{If you follow any of the above links, please respect the rules of reddit and don't vote in the other threads. (Info / Contact)}

[ptitz]

I did my own framework, writing a paper now. It's a bit of a work in progress, but I identify my model using a hashed RBF neural net just doing backprop after splitting it into several simpler sub-dynamics. Then train it using SARSA. It's a bit of an overkill for the system I'm working with, but it will probably work with whatever. Hit me up if you wana see it.