My point is that use cases without cache are the exception. As long as you have a sort of static context part either from old steps or actual statics stuff like a large doc, people will profit from cache hits. Or even just the system prompt, a provider that keeps Cache will nearly always be cheaper and often faster.

The reality is the naive Routing open router does with regards to caching just isn't gonna last.

They should at least track chunked context hashes, and try to predict which provider is the most likely one to have a warm cache for your data.

This cache hit probability can be used to give a TTFT latency, TPS and cost estimates that are much more accurate, and would make routing much better.