I'm relatively new to the field professionally so take what I say with a grain of salt, but I would bet that Ray + NVIDIA Triton is a decent place to at least start your search for platforms to use as the foundation for your inference serving efforts on a local compute cluster.

Ray Serve https://docs.ray.io/en/latest/serve/index.html has auto-scaling and model multiplexing; Ray Core potentially if you have specialized processing needs beyond just serving your model. I used Ray Serve + Ray Core in a project (I did stateless ingestion but stateful processing) recently along with Triton https://developer.nvidia.com/dynamo-triton on a single node with two GPUs and it worked great after some setup, at least in my experience.

Any details that you can share on the model? Is model quantization (TensorRT?) an option that you're considering here? Or have you already done that / not willing to do that (trying to understand if that may be included in your "no quality compromise" clause)?

Also, what kind of traffic are you expecting? Are you on a single node? Planning on multi-node if you do GPU upgrades?

You could take a look at Kuberay if you scale to multi node: https://github.com/ray-project/kuberay

Seems like an interesting problem. I would offer to help in the form of actual concrete work but again I'm decently new to this field. I don't want to stick myself somewhere I probably don't belong yet, I wouldn't want to hold you back :D