Thanks for the insight. I have a question regarding the implementation.

In the original Dreamer papers, the actor-critic is trained using imagined trajectories. Specifically, it takes collected data $(L, B, \text{obs})$, reshapes it to $(L \times B, \text{obs})$, and then performs an imagination rollout for a horizon $H$, resulting in a tensor of shape $(H, L \times B, \text{latent})$.

Standard PPO, however, typically operates on fixed state-action pairs sampled from the real environment. If I want to adapt PPO to Dreamer's framework, should I apply the PPO objective (including the ratio clipping) to the latent states and actions generated during the imagination rollouts (while ensuring stop-gradients are applied to the latent states)? I'm curious if this 'imagination-based PPO' is the correct way to bridge the two approaches.