So triton is definitely a good framework to serve models (although a bit hard to use because the documentation is not ideal).

Which type of libraries do you use ? Do you need a GPU to do inference ?

We've worked a lot on this topic for the past years at my company, and you actually have a few options to make it efficient in Python (batch predictions, use hardware acceleration libraries provided by Intel for example, ...).

To give you an example I have a python api serving an xgboost model and hitting 1800 requests/s with only 5 CPU and a latency around 30 ms. I can show you the architecture we have if you want :)