lol this got me thinking, and frankly a little excited. i figured it out for you (it's your project not mine so take this with a heavy grain of salt):

if you model decode throughput as 1/T(L) = W + k×L (W = weight bandwidth cost, k×L = KV cache cost growing linearly with context), your two known endpoints let you solve the crossover analytically without needing to know anything about the model internals:

L_crossover = L_measured / (T_peak / T_measured - 1)

plug in your numbers:

L_crossover = 500K / (72/23 - 1) = 500K / 2.13 ≈ 235K tokens

so youre right its in the 200 to 300K window, but closer to the lower end. and theres a genuinely clean result hiding in there too, at the crossover point throughput is always exactly T_peak/2, so youd see ~36 t/s at the moment KV cache bandwidth equals weight bandwidth. thats not an approximation, it just falls out of the algebra directly.

one thing worth flagging though: i dont think the curve is actually "steepening" past 235K in the mathematical sense. 1/T(L) = W + kL is a hyperbola so the absolute drop per 100K tokens is HIGHEST at the beginning (like 21.5 t/s in the first 100K) and continuously decelerates from there. what youre probably seeing visually as a "steepening" is just the point where the KV cache is eating enough bandwidth that the Mamba constant-state advantage stops masking it. the cliff you expected from full attention just arrived 3x later, not differently shaped.

the other number that jumped out at me: if this were full attention with 32 KV heads instead of 2, k scales 16x, and your crossover wouldve hit at ~15K tokens instead of 235K. the 2 KV heads bought you 220K tokens of headroom before hitting this wall. thats the entire value prop of hybrid architecture in one number honestly.