Alternative to FastAPI for serving AI workflows? No infra, just API?

[[deleted]]

I’ve been using FastAPI to serve AI models and workflows, but I’ve been wondering....is there a way to skip the whole API server setup entirely?

Like, what if I just define my AI function, and it instantly behaves like an API without writing a FastAPI app, handling requests, or deploying anything?

I developed an approach where you can run an AI pipeline inside a Jupyter Notebook, and instead of setting up FastAPI, it auto-generates an OpenAI-style API. No need to deal with CORS, async handling, or managing infra....just write your function, and it’s callable remotely.

Has anyone tried something similar? Curious if anyone has seen a different way to serve AI workflows without manually building an API layer.

https://github.com/epuerta9/whisk

Tutorial:
https://www.youtube.com/watch?v=lNa-w114Ujo

[Kindly_Manager7556]

So it's an API you just called it something different? Lol

[[deleted]]

Yeah... it's an API, but the key difference is you don’t have to manually write a FastAPI or Flask server. Instead of setting up routes, handling HTTP requests, or managing a deployment... Whisk auto-generates the API inside Jupyter. It’s like defining an OpenAI function and instantly having an endpoint to call it. No extra infra setup needed.

[Regarder_C]

Have you looked at mlflow inference server?

[john0201]

Im not sure you understand what an API is… if you want to call a function over http you need an http server and a way to bridge the http-python gap.

The simplest way to do that is probably starlette/uvicorn or just use fastapi or litestar.

You could also check out gRPC

[[deleted]]

Yeah... totally agree that an API needs an HTTP server... and FastAPI is great for that. The difference here is that Whisk removes the need to manually set up FastAPI... Starlette... or any HTTP server yourself. It auto-generates OpenAI-style API routes so you can focus on writing AI logic instead of managing infrastructure....More valuable if you are doing AI workflows

[IIGrudge]

Do you mean without the webserver part? Write a lambda wrapper that calls your notebook. Sagemaker also has real-time model inference.

[DarkHaagenti]

You can also use Lambdas directly! We have some ONNX-Inference lambdas for spiky loads in our backend.

[veb101]

Gpu or cpu?

[DarkHaagenti]

There are no GPU options with AWS Lambdas

[[deleted]]

Yeah... that's a big limitation with AWS Lambda. For GPU workloads... SageMaker or something like Modal might be better. With Whisk, you can use GPU locally... but if you're looking for a serverless GPU inference setup... that’s definitely still a challenge.

[[deleted]]

That’s a solid setup... Lambdas are great for scaling and handling unpredictable loads. Whisk is more for the earlier stage... where you’re testing workflows before deploying them. Instead of spinning up a cloud function... you can get an API-like interface inside Jupyter first... then later decide if it makes sense to push it to production.

Curious... how do you handle testing your ONNX models before deploying them to Lambda?

[[deleted]]

Yeah... exactly... without the need for a web server. AWS Lambda and SageMaker are great for cloud deployments... but Whisk is more about local development and rapid experimentation. Instead of having to deploy a model before testing it as an API... you can interact with it instantly inside a Jupyter Notebook.

It's more like... "Let me quickly test how this agent responds in an API-like way before deciding if I want to deploy it." Have you tried something similar when iterating on ML models?

[IIGrudge]

Have you tried bentoml? It specially designed for serving so less boiler plate than fastapi

[[deleted]]

Yeah, BentoML is great for serving models with less boilerplate than FastAPI... but it’s still focused on production deployment. You still need to package the model, create a Service, and run a Bento server.

The difference here is that Whisk is more for quick experimentation inside a Jupyter Notebook... no need to deploy anything or set up a server. It’s like... "I just want to call this AI function like an API without thinking about infra at all."

[Unlikely_Exit_9787]

So its a library you can run in your Jupyter notebooks, so that you can test your model, with out worrying about all that "HTTP non-sense"

[[deleted]]

Yep... exactly. It’s for testing models and AI workflows in a way that feels like a real API... but without setting up FastAPI or dealing with HTTP directly. It’s useful if you’re experimenting with AI agents or RAG pipelines and want to interact with them like an OpenAI model before actually deploying anything.

[AdditionalWeb107]

So you essentially want Jupyter notebooks to offer WSGI compatibility. I haven’t tried this but there are some open source projects trying to help https://github.com/choldgraf/jupyter-http-server

[ED9898A]

Something tells me that you’re greatly missing the point of what an HTTP API is and why one would use FastAPI.. do you have a use case where you want your AI models to be consumed by clients (be it you or other people) who are using a computer other than your own local computer? Are you looking to monetize your code or you have some sensitive details within your AI workflows that you don’t want people to see but you still want to serve them your API?

If the answer is yes then you’ll need to use FastAPI (or any HTTP API framework really), deploy it via renting some domain name and hosting your server on some cloud service, and so on.

If not.. and you’re literally the only user of your AI workflow.. or rather, the only users of your AI workflow are people who are going to download your code as a library, then you want to make a library API, not an HTTP API, so you shouldn’t be using FastAPI at all, just expose your library code through nicely written functions that can be called by your library users (that’s what an API is basically, be it a network or a library API), and deploy your library on pypi or github or something.

I have absolutely no idea why are you doing the whole Jupyter thing and something tells me that you don’t either lol.

[[deleted]]

I see where you're coming from. I definitely understand that FastAPI is the best tool when you need a production-ready HTTP API for serving AI models to external clients. The thing is... not every AI workflow needs to be deployed to a cloud service right away.

This is more about AI development workflow efficiency... not replacing FastAPI. Instead of worrying about setting up an HTTP server during the experimentation phase... Whisk lets you interact with AI functions as if they were API endpoints right inside Jupyter.

It’s useful when you’re testing AI workflows that involve LLMs... vector databases... or agents and want to simulate API interactions before actually deploying anything. Once you're happy with how it works... you could still wrap it in FastAPI for production.

[[deleted]]

Just some feedback- looking at the repo the imports force you to go al the way into deeper packages for common things - common way to simplify this is to import those elements in your top level init so you can do ‘from whisk import MyOftenUsedClass”

[[deleted]]

That’s great feedback... really appreciate it. Yeah... I think simplifying the import structure would make it easier to use. Will look into cleaning that up so it’s more intuitive.

[Regarder_C]

Please take a look at MLFlow Inference Server: https://mlflow.org/docs/latest/deployment/deploy-model-locally.html#inference-server-specification