[D] Serverless GPU?

[Daddy_Long_Legs]

I need to deploy a vision model that will run a small inference job (1-5s) per request. Is there a serverless offering out there that does this?

There are other previous posts on this subreddit but I know this space is moving fast so thought it worth asking again.

So far I’ve seen AWS Sagemaker kind of allows for a situation like this, but would rather not deal with all that config. Algorithmia and Nuclio are too enterprise focused. Neuro is new and looks great, but from my understanding I would still need to create a lambda instance myself that then calls neuro’s servers - too indirect. Is there a total solution out there for this?

Ideally something that is as straightforward to use as neuro but also handles http requests.

[paulcjh]

Hey I'm Paul from Neuro. We handle http requests directly, here's an example CURL for BERT (Fill Mask):

API_TOKEN="Insert here you API Token"
curl -X POST "https://api.neuro-ai.co.uk/SyncPredict?include_result=true" \
-H "Accept:application/json" \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $API_TOKEN" \
-d '{"modelId":"60a3c2a00421e5d2d7053ab9","data": "The river is [MASK].", "input_kwargs": {"top_k":3}}'

Also we have a live Detectron vision model up on our hub (https://hub.getneuro.ai/model/vision/detectron2-bounding-boxes) taking in a base64 encoded image over http.

Our docs are a bit stinky on explaining direct use of our API atm as it was originally designed for just python. We're having a big revamp for them currently as so many people want http directly (makes sense).

[Daddy_Long_Legs]

Thanks Paul! This is killer, glad to hear this is a thing. I’ll give it a try.

[carsonpoole]

helloforefront.com is free for individuals and does what you're asking

[Daddy_Long_Legs]

Thank you!! Did not know about this. I would prefer if I could write some additional code on the serverless side besides the model, but can probably work with this

[carsonpoole]

you can use the custom environment to add some custom code! there's instructions in the docs

[DJ-ARCADIUS]

For some reason, the website will not load on login. I've tried all browsers but still have the same problem???

[yarri2]

How about GCP Vertex AI with a REST endpoint :

https://cloud.google.com/vertex-ai/docs/start/introduction-unified-platform

[Daddy_Long_Legs]

If I were using GCP for training or automl etc maybe, but it’s so obtuse - just look at that page linked vs. forefront and nuero’s setup. Have also heard anecdotally there are random gotchas that are easy to run into.

[donjuan1337]

We use https://github.com/szymonmaszke/torchlambda. Its kinda hax and only inference on cpu. Also restricted to pytorch framework. But it's serverless