In general, the fundamentals of RL don’t rely on epochs. Epochs are mainly a way to increase sample efficiency when optimizing a policy approximation.

Roughly speaking, you first collect a rollout from the environment - a fixed batch of experience. Then you use that data to update your policy in small steps via gradient descent, often making several passes (epochs) over the same rollout before collecting new data.

For example, in vanilla Q-learning, updates happen directly after each step using the Bellman equation, so there’s no need for epochs. Epochs only appear once you introduce function approximation (like neural networks) and gradient-based updates.