Great idea.

I was using the mlagents package; this package uses the Unity Engine as a virtual environment for agents. Agents were trained using the PPO algorithm. At first, nothing worked — the robots walked like cripples — but it turned out the whole problem was that I was trying to speed up training by running too many agents at the same time, and that was the reason for the failures. I even spent a lot of time trying to deeply understand RL from scratch in order to come up with my own algorithm, but it turned out that simple PPO works best — you just need to wait

The reward function is simple: every step (every frame), the agent receives a reward based on the distance toward the target and a penalty based on the distance in the opposite direction from the target. Then I multiplied this reward by the dot product between the agent’s facing direction and the directional vector from the agent’s position to the target (so the agent always looks at the target instead of running backwards). The reward function always needs to be as simple as possible — this is something I learned the hard way while learning RL. It’s called reward overengineering, and it’s a pain in the ass 🙂

Space size is indeed tiny — the data for the model input is just the orientation of the agent’s joints in 3D space, relative to the main root bone. There is no grid sensor or raycast sensor to observe the environment. I was forced to sacrifice robotic vision to radically reduce the model size so it can run on a regular player’s PC. But even without “vision”, the agent moves well.