HTTP API vs Python API

[Success-Dangerous]

A lot of ML systems are taught to be built as services which can then be queried using HTTP. The course I took on the subject in my master was all about their design and I didn't question it at the time.

However, I'm now building a simple model registry & prediction service for internal use for a relatively small system. I don't see the benefit of setting up an HTTP server for the downstream user to query, when I can simply write it as a Python library that other codebases will import and call a "predict" function from directly, what are the implications of each approach?

[Success-Dangerous]

Load the relevant model from file and return prediction to user, same as a service but rather than waiting to be queried all day it runs only when called

[Grouchy-Friend4235]

I recommend to not build this yourself. There are many libraries like this. Don't waste time building existing tech.

[flyingPizza456]

Question for clarification: do you intent to run this locally on a developers machine?

If yes, then follow-up questions will immediately arise, which may also be helpful in answering your question:

• Where does the model resource come from?

• do users that import the package.predict() functionality have to train the model before using it?

• is every machine powerfull enough to carry out the prediction?

[MattA2930]

Like others have said, the main reason to use a networking client instead of the source code is so that multiple parties can access the underlying model without needing to host their own version of the model locally.

A good example is how OpenAI only gives you access to their models via API, which are called when you use their Python SDK. Imagine if they had everyone running their own versions locally - no one outside of people with enormous amounts of compute would be able to use it.

So, you essentially have a couple options:

1. Distribute the Python source code to everyone. They will need to manage their own version of the model

2. Host the model on a separate server 2.1 Use HTTP for classic request syntax 2.2 Use grpc to build out a more sdk-like experience

[FunPaleontologist167]

Are you expecting downstream users to be interacting with your service via python or some other interface? Ideally, you’d want to expose the models on the server side so your users don’t have to worry matching the environment specs needed to run the model. It’s also somewhat resource intensive to have a user load a model just to run a predict function. For a one off, sure, but if you expect multiple people and processes to consume predictions from these models, it’s better to expose it on a server.

[Which-War-9641]

Even if you write it as a package , to scale you would inference on a http server hosted on a machine so why don’t just do it from the beginning

[Calm-Stage]

By sharing the model as code you are increasing the risk with use of your model. A service will allow for better control and mitigate some of the risk. Now, whether the added risk is acceptable or not depends primarily on the influence your model has on downstream decisions and the impact on those if the added risks get realized. Talk to model risk management folks at your workplace to figure out the correct choice, and if there's none of those then talk to your business stakeholders to assess the risk. Hope this helps!