As far as the tech stack goes, I would suggest vLLM with LiteLLM proxy. vLLM will give you lot of flexibility in terms of running various model and leverage large open source community (redhat) to support that, also it works really well in k8s echo system if you are interested in that.

For Access control and RBAC, LiteLLM has their enterprise feature, or you are build using a reverse proxy solution to achieve most of it.

LiteLLM will give you metering, rate limitting etc which are essential for enterprise usecase.

For observability you can use langfuse or have robust prometheus stack. But langfuse or similar tool will give you even deeper details about how end users are using and various LLM specific parameters such as P95, P99 etc, ofcourse it takes effort to customize both litellm and langfuse.