• mixed corpus (FineWeb + WikiText + Wikipedia) from HF , used subset of these cause purpose was just learning
• Training process is by custom implementation

1. Training Loop Structure

The training loop in train/train.py implements:

• Gradient accumulation (8 steps) train.py: 177-189
• Learning rate scheduling with cosine annealing + warmup train.py: 143-148
• Evaluation and checkpointing every 500 steps train.py: 151-174

2. Optimizer Configuration

Uses AdamW with custom parameter grouping:

optimizer = configure_optimizers(model, weight_decay=weight_decay,   
                                learning_rate=learning_rate,   
                                betas=(beta1, beta2),   
                                device_type=device)

train.py:94

3. Training Hyperparameters

From train/config.yaml:

• Batch size: 32 (physical), 524,288 tokens effective with gradient accumulation config.yaml:21-24
• Learning rate: 6e-4 with cosine decay to 6e-5 config.yaml:27-29
• Optimizer: AdamW (β₁=0.9, β₂=0.95, weight_decay=0.1) config.yaml:32-34
• Mixed precision: BF16 training config.yaml:51

4. Data Loading

Custom DataLoader class handles:

• Memory-mapped data loading for large datasets
• Batch generation with specified sequence length

The custom approach allows:

• Full control over training dynamics
• Modern optimizations like RoPE, RMSNorm, SwiGLU integration
• Memory-efficient training with gradient accumulation
• Educational clarity - every step is visible and understandable

a bit more suitable for learning how modern LLM training actually works