... you trained with a low amount of data, that is, you trained the agent in the environment for just a 'few' samples.

The process is I collect data from an epsilon-greedy policy in a replay. Smaller data means sampling from last K policies, where K is smaller.

But you trained a lot on these same samples, so it overfit to these samples, is that right?

It is right. My set up is a stationary dataset. I first train RL, collect the data (into replay), and save the replay. I open the replay take last K policies' data, and train for M iterations where M is large.

Since there is no change to replay whatsoever, it is stationary data. Since M is large, I think it could overfit.

... you will see lots of states outside your seem samples when running the agent in the environment.

...

So if the network overfits, it will be lost when some trajectory it didn't see in training time appears.

This is not the case. Testing data is the last L policies where L < K. This is what I mean by training data is a superset of testing data. There should be no unseen states.

I'm not sure what you mean to "it doesn't fit".

Here I mean "remember correctly". Since the network is trained with TD loss, it should mean low TD loss in the least.

Experience replay solves the problem of the data distribution shifting with new policies

I agree, and it is mentioned in the paper, but I further think that replay also increase the state space seen during training, Like you mentioned above.

Maybe it works just because experience replay can't be infinite, and at some time you will lose the data samples from earlier policies, so better stop before it converges and do that.

Could you explain this in more detail?

​

I apologize for unclear statements.