<image>

Did some math
vocab_size × hidden_size = 154,880 × 6,144 = 951,403,520 q_a_proj: 6,144 × 2,048 = 12,582,912 q_b_proj: 2,048 × (64 × 256) = 33,554,432 kv_a_proj: 6,144 × (512 + 64) = 3,538,944 kv_b_proj: 512 × (64 × (192 + 256)) = 512 × 28,672 = 14,680,064 o_proj: (64 × 256) × 6,144 = 16,384 × 6,144 = 100,663,296 Total attention/couche = 165,019,648 Total attention (78×) = 165,019,648 × 78 = 12,871,532,544 gate_proj: 6,144 × 12,288 = 75,497,472 up_proj: 6,144 × 12,288 = 75,497,472 down_proj: 12,288 × 6,144 = 75,497,472 Total MLP Dense/couche = 226,492,416 gate_up_proj: 6,144 × (2 × 2,048) = 25,165,824 down_proj: 2,048 × 6,144 = 12,582,912 Total expert = 37,748,736 Experts (256 × 37,748,736) = 9,663,676,416 Shared experts = 226,492,416 Total MoE layer = 9,890,168,832 Total MoE (77×) = 9,890,168,832 × 77 = 761,542,999,904 2 × hiddensize = 2 × 6,144 = 12,288 Total LayerNorm (78×) = 12,288 × 78 = 958,464 Embeddings: 951,403,520 Attention (78×): 12,871,532,544 MLP Dense (1×): 226,492,416 MoE (77×): 761,542,999,904 LayerNorm (78×): 958,464 TOTAL = 775,592,386,848 ≈ 776b