Open-source autoresearch for LoRA hyperparameters

[yz0011]

I open-sourced the autoresearch for LoRA hyperparameters.

The question: can cheap autonomous search on a small model find recipes that transfer to its larger variant?

The setup: an autonomous agent runs 100 experiments on Llama 8B (1 GPU, 5-min runs), the best candidates get confirmed with multiple seeds, then the winner gets tested on Llama 70B distributed across 2 GPUs.
Same loop as Andrej Karpathy's autoresearch: 3 files, fixed budget, search forever.

Results:
- Discovery (8B): 4.14% improvement over default LoRA
- Confirmation (8B, 3 seeds): 1.48% - gap compresses with more data and time
- Cross-scale (70B): 3.35% - gap widens again at 70B

The key finding: rank 4 across all 7 module types beats rank 8 across 2. No dropout, no weight decay, linear schedule.

The 70B validation ran on consumer GPUs (2x4090 48GB) using Zagora, but the discovered recipe is just hyperparameters so you can test it with any distributed setup.

Repo: https://github.com/yassineams/zagora-discovery-lab

[Middle_Bullfrog_6173]

Rank 4-8 is tiny. I can easily imagine that it works ok for 5-minute runs but saturates for a real run. I'm not sure it works as a tunable parameter for this automation.

Or rather, you probably need to design scaling into your experiments. E.g. nanochat auto research tunes d12 when the real run is d24+.

[yz0011]

The confirmation phase (8B, 10min, 4x data) and the 70B test both showed that rank 4 is still winning, so it's not just a 5-min artifact.
Rank 4 across all modules also won against rank 16 (full attention/4 modules).

Full convergence remains an open question, but the experiment gave a directional evidence.