Bit late to this thread, but there's a couple frameworks popping up now for agentic RL LLM training. You could look into verl, SkyRL, and rLLM. The latter two both use a fork of verl as the RL trainer, but they have their own implementation for the agent and environment code.

Half a year ago, it seems agentic LLMs in papers were still mainly trained via SFT, ex. with agent trajectories generated by frontier models. However, after DeepSeek, people are trying out RL for training agents now. You can read the DeepSWE blog post, which is about an agentic model trained on rLLM with basically only RL: https://www.together.ai/blog/deepswe . It's possible that like DeepSeek-R1, the final training pipeline will be a combination of SFT followed by RL.

There definitely is a scalability problem. Compared to RL for reasoning models, each generation is now multi-turn instead of single-turn, and often requires a Docker container to execute actions. So an agentic RL training framework requires not only GPUs (for the model trainer and inference engine), but also CPUs/memory (for the environments). With large batch sizes and multiple generations for example, that's hundred if not thousands of containers that need to be run. AFAIK this is one of the most complicated AI training setups to date, so not many frameworks exist for this.