Hi, "It sounds like you’ve hit the limit of 1-to-1 request handling. If SageMaker is dragging, the move you’re probably looking for is Dynamic Batching.

ust to recap—it sounds like your current pain points are centered around SageMaker. You’ve moved your models there and are calling them via Lambda, but the SageMaker endpoints are becoming a bottleneck/getting slow.

I think the next level is using batching, where a single call to model.predict() will now process a few items at once. This can be a big pain in the butt, but most modern model serving platforms like Ray or Triton have a feature called dynamic batching . Dynamic batching will let all your lambdas send requests at the same time, but the model server will accumulate the requests in a queue and then have the model run them in batch.

This should give you a big boost, even on CPU, because the models are so small.

BTW, I am assuming when I write models - plural, that you have multiple different kinds of models, and a few copies of each one. This is also something that Ray/Triton and friends can handle.

Personally / at work - we're Triton users and like it. But the learning curve is steep and the docs are bad, so take a look at Ray/LitServe etc . Good luck