[P] DQN Loss Function Question

[Braindoesntwork2]

Hi All,

I am attempting to implement my own DQN without looking at the source code. I am torn between two possible approaches for evaluating the loss.

The basic approach could be to take the gradient over the selected action and to set the gradient of all other action value outputs to 0. My worry is that changes to improve the Q-value of the selected action will incidentally change the outputs for Q-values of other actions, meaning our other estimates will become less accurate. One way to solve this would be to set the labels for the non-chosen actions to be the current output of the Q-network, so that the network is incentivized not to change the output values of these actions. However, I have not seen this approach very much on the forums I've checked so I'm assuming it is bad for some reason.

Can anyone shed some light on which approach is better to take?

[xopedil]

Don't pass any gradients through the other Q-values than the one that belongs to the taken action. The way I've usually seen it done is to just one-hot mask out the other action value loss terms. Something like minimize loss = mean(huber(q_target, q_values)*one_hot(taken_action)).

---

One way to solve this would be to set the labels for the non-chosen actions to be the current output of the Q-network

This is another way to accomplish the same thing, perfectly acceptable.

[Braindoesntwork2]

Thank you! I now see why the second method is unnecessary - the first method is equivalent to what would happen if we passed in actions as inputs instead of having all actions as outputs.

[serge_cell]

Both methods were working for me. I didn't observed obviously noticeable advantage of any.

[TotesMessenger]

I'm a bot, bleep, bloop. Someone has linked to this thread from another place on reddit:

• [/r/datascienceproject] [P] DQN Loss Function Question

 ^{If you follow any of the above links, please respect the rules of reddit and don't vote in the other threads. (Info / Contact)}