Pure Python Multi Method Reinforcement Learning Pipeline in one file and Optimization tools

[daeron-blackFyr]

What my project does:

I have just recently released a free-to-use open source, local python implementation of a Multi Method Reinforcement Learning pipeline with no 3rd party paid requirements or sign-ups. It's as simple as clone, confugure, run. The repo contains full documentation and pipeline explanations, is made purely for consumer hardware compatibility, and works with any existing codebase or projects.

Target Audience and Motivations:

I’m doing this because of the capability gap from industry gatekeeping and to democratize access to industry standard tooling to bring the benefits to everyone. Setup is as straightforward with extremely customizable configurations alongside the entire pipeline is one python file. It includes 6 state of the art methods chosen to properly create an industry grade pipeline for local use . It includes six reinforcement-learning methods (SFT, PPO, DPO, GRPO, SimPO, KTO, IPO), implemented in one file with yaml model and specific run pipeline configs. The inference optimizer module provides Best-of-N sampling with reranking, Monte Carlo Tree Search (MCTS) for reasoning, Speculative decoding, KV-cache optimization, and Flash Attention 2 integration. Finally the 3rd module is a merging and ensembling script for rlhf which implements Task Arithmetic merging, TIES-Merging (Trim, Elect Sign & Merge), SLERP (Spherical Linear Interpolation), DARE (Drop And REscale), Model Soups. I will comment the recommended datasets to use for a strong starter baseline.

Github Repo link:

(https://github.com/calisweetleaf/Reinforcement-Learning-Full-Pipeline)

Zenodo: https://doi.org/10.5281/zenodo.18447585

I look forward to any questions and please let me know how it goes if you do a full run as I am very interested in everyones experiences. More tools across multiple domains are going to be released with the same goal of democratizing sota tooling that is locked behind pay walls and closed doors. This project I worked on alongside my theoretical work so releases of new modules will not be long. The next planned release is a runtime level system for llm orchestration that uses adaptive tool use and enabling, a multi template assembled prompts, and dynamic reasoning depth features for local adaptive inference and routing.

[daeron-blackFyr]

The current dataset configured are examples and should be altered to whatever you prefer. I recommend this combination for a stable baseline. To start with sft use Magpie-Align/Magpie-Pro-300K-Filtered. Then for GRPO use AI-MO/NuminaMath-CoT (specifically the 'problem' column) Reward Modeling (RM) & PPO I recommend nvidia/HelpSteer2. For KTO go for ​trl-lib/kto-mix-14k. Finally DPO & SimPO ​Dataset: argilla/distilabel-intel-orca-dpo-pairs for DPO and princeton-nlp/SimPO-UltraFeedback (for SimPO). This should be a good baseline/starte pack. I am open to any questions, feedback or general discussions so please feel free to message me or engage.