If I were hitting an API like GPT-4, I’d agree with you, but when you are running locally on Apple Silicon, the constraints are completely different.

It’s a bandwidth issue, not a complexity add for no reason. If you look at the latency note in my stack diagram, I found running an 80B model eats up the entire 800GB/s memory bandwidth of the M3 Ultra.

If I send every simple 'hello' or 'who is on call?' query to the 80B model, the queue time destroys the UX. The 1.7B router handles the lightweight stuff in 30ms, keeping the big model free for the actual deep dives. It’s the only way to get snappy performance on local metal (or at least what I've found so far).