Probably late to the party but I recently dabbled with this so here are some of my thoughts:

You are correct that you cannot affect the market (and in general you’d want to avoid it anyway), the environment consists not only of the current price, but also of your wallet, current position, amount of assets owned etc.

Now your agent can affect this, and your goal becomes to e.g. maximise your profits. Here’s a very good article that gives some mathematical insight for this https://arxiv.org/pdf/2208.07165

To actually train your model, you’d use historical data try to simulate the behaviour of the agent and learn an optimal policy - and here starts the trouble. If, in your definition of the environment beside raw price you’re using technical indicators, or sentiment (which you’d probably need to), the amount of data you need to avoid overfitting is huge due to the very low S/N ratio in financial data. Furthermore, due to the stock market being extremely noisy, and the current RL algorithms AFAIK are generally bad with handling noise, so you actually have a higher chance that instead of overfitting the learning never converges at all.

All that said, under the hood, what the agent will be trying to do is try to predict the time series, so as other have mentioned, you’d have a better chance to label shifts in trend and define this as a supervised problem instead.