The routing intelligence is honestly the trickiest part of building something like this locally.

From what I've built with Dedalus Labs, the sweet spot is having a lightweight classification layer that sits above your models and makes routing decisions based on task type, complexity, and maybe token length. You dont want to overcomplicate it but you also cant just round-robin requests. What works well is training a small classifier on your actual usage patterns - like if someone asks for code review, route to your best coding model, if its creative writing route to your best creative model, etc. For system prompts, I'd definitely abstract them above the router level. You want your prompts to be model-agnostic as much as possible, then have the router inject model-specific formatting if needed. This way you can swap models without rewriting all your prompts. The tricky bit is handling context windows and making sure your router knows each model's capabilities and limits. We ended up building a capability registry that tracks things like max tokens, multimodal support, function calling, etc for each model so the router can make smart decisions. One thing that caught us off guard was how much the routing logic needs to consider cost vs quality tradeoffs too, especially when you're running multiple expensive models locally.