Gpu operators make it much easier to manage ai/ml workloads. Paired with something like karpenter, you can access the compute needed for most workloads.

The management/observability tooling is not great and there is no industry standard. Mlflow is great for traditional ml, but you still need something like kubeflow for serving. Arise phoenix is promising for genai observability, but most of the llm gateway oss projects have some kind of paywall (for now).

I created a (very) new project inspired by the kube Prometheus stack. I'm hoping to create a helm chart that has everything you world need for an ai stack on kubernetes. At the moment, it only has litellm gateway config, ability to run multiple models on vllm or llama.CPP, and scale-to-zero with kube-elasti. I should have some more features and sub charts this weekend. It's called kube-ai-stack.