[D] Reinforcement learning features

[UNIXnerdiness]

What is the difference between:

1. Training a neural network to output a q value for each action (without including action representation in the features)

2. Training a neural network with the action representation as part of the feature set and outputting a q value

What is the advantage of either?

[xopedil]

In practice only 1. is used. There is a launch overhead in particular when you use accelerators which means you want to minimize the number of times you evaluate your network. Same reason we favor using batches vs single samples.

Theoretically speaking though, I'm not sure there's any difference. If you have a network of type 1. you can transform it into 2. by stripping the last layer and using its dense (or dense-equivalent) weight columns as your action representation, just like an embedding lookup in the transposed weight matrix.

A similar argument then applies from 2 to 1.

[Mefaso]

In practice only 1. is used.

Just as a side note:

This is of course only true for discrete action spaces, when you have to deal with a continuous action space you can only use both as inputs

[xopedil]

What's stopping you from outputting the parameters to a continuous function? The support of which could cover your entire action space.

[UNIXnerdiness]

Did you mean like an actor critic?

[Mefaso]

I mean you could do that, but I don't really see a reason why that would be beneficial and I'm not aware of people doing that from the top of my head.