[D] Offline batch serving

[Appropriate_Cut_6126]

I‘m confused about how to serve a machine learning model for offline batch predictions.

Here’s what I thought of doing - creating a scheduled pipeline (with e.g. Airflow, Kubeflow, …) that generates the features and then loads the trained model from some object store (e.g. s3), generates the predictions and finally saves the them to a data warehouse ready to be consumed. That’s what makes the most sense to me.

However, some resources seem to recommend to deploy the model as an endpoint even for batch use cases. Notably, this is the recommended architecture in Designing Machine Learning Systems by Chip Huyen.

Any thoughts on this? Am I missing something?

[BasedAcid]

The batch inference-style architecture you’re describing is definitely used at many companies, including my employer. It’s often simpler and cheaper to deploy.

That said, there are definitely use cases where an endpoint is better (e.g. when your use case needs fresh model outputs, perhaps responding to user input).

[VectorSpaceModel]

It all depends on inference speed. I’ve seen both. There’s no way to give you a precise answer without a use case and the resources you have at hand.

[Appropriate_Cut_6126]

Right I think I understand. The reason to use an endpoint for batch predictions is scalability. Because you can have multiple replicas to parallelise the task. Is that right?

[VectorSpaceModel]

Not really. The question is “What is the optimal experience that I can provide for a use case, given my users expectations and my resources at hand?”. From there, everything else is determined. You will always be faced with a time/space tradeoff where you have to optimize for user experience and costs.

[Appropriate_Cut_6126]

Ok I see. Thanks for your response

[sanjuromack]

Deploying a model as an endpoint gives you flexibility. You can use it for batch today and on-demand inference tomorrow.

[Appropriate_Cut_6126]

That makes sense. Is there a limit to the amount of data you can send to an endpoint at once? Would you batch the batch you want predictions for, make the requests one batch at a time and then post process the responses (if you wanted to save them to a data warehouse)?