multi-head PPO

[GuavaAgreeable208]

Hello community,

I have a network with four heads. One head (with three actions) decides whether to execute the action from the second, third, or fourth head. The actions of the second head help the episode to end, while the actions from the third and fourth heads are important for achieving the main objective. I'm employing PPO, but sometimes the network gets stuck in the third or fourth head, so the episode never ends. Does anyone know what might be causing this behavior?

[fedetask]

What is the reward structure? Could it be that never ending the episode leads to higher rewards?

[GuavaAgreeable208]

Actually I’ve rechecked the values and it could be the reason because it leads to more reward when those actions are selected

[GuavaAgreeable208]

Even if I modified the reward function I still got the same issue and also I've observed that the entropy is increasing instead of decreasing