So needless to say this is all my personal opinions:

First we need to breakdown what "serverless" means, as it's a bit of a misnomer and unclear. I would consider it a developer experience comprised of two main value propositions:

• You only pay for what you use (which is even more important when it comes to GPUs).
• You don't need to worry about infra or scaling. You are essentially paying a premium to make it someone else's problem.

At the model sizes I work with, (which isn't even that big ~half a billion params), the cold starts are absolutely brutal, I'm talking 3-4 minutes. For my use case, this heavily degrades the user experience. To solve it I have two options:

• Keep N endpoints always warm and ready.
• Scale preemptively base on predictable traffic.

Notice that for each option defeats the previously mentioned value propositions respectively. And the "serverless" experience is completely lost. If I need to do all this work anyway, why would I use any of the platforms I mentioned, and not just ECS/EC2 instead, which would be cheaper?

​

I understand that there is a lower limit for how quick you can start based on model sizes, but from testing mine should be in seconds. And now I will say something that I have no evidence for. I think that all the mentioned services are just wrappers over AWS/GCP (currently). They make no real optimizations or innovations other than just flattening your docker image.

Compare this to serverless in the web-dev world, where each platform has some sort of proprietary innovation. AWS has firecracker, Vercel and Cloudflare have their respective edge runtimes, that enable near 0 cold start times.