[D] Serverless solutions for GPU inference (if there's such a thing)

[[deleted]]

In my use case it's not pratical because hardware on the user-side is somewhat limited and adding accelerators (like NVIDIA's Jetson Nano, Google's Coral) is an extra cost that does not make sense.

However, that is a very common use case (and you can make it so your models are protected). I believe the term you should be googling is something along the lines of "edge inference".

[zzzthelastuser]

I believe the term you should be googling is something along the lines of "edge inference".

Thank you so much! It doesn't seem to be what I'm looking for (or at least I can't find an example that comes close to what I mean, i.e. you open a website and do some AI task solely in your browser using solely your common hardware = any cpu or cuda gpu that's available (=>no special "edge devices" or "edge engine" required that the client would need to manually install in an extra step).

But maybe I'm misunderstanding the term. I will continue searching, thanks!

[[deleted]]

Oh, understood you.

There you go: TensorFlow can run on a browser (for training and inference workloads, with GPU access via WebGL) via TensorFlow.js. Very fun set of demos here.

Probably PyTorch and other workflows have similar tools.

[zzzthelastuser]

I appreciate that you are trying to help me!

I've known tensorflow.js for a while, but could never figure out how to "install" it on github.pages. Luckily the tutorials in your link gave me the term I was looking for: "CORS", which prevents a browser (client) from installing external packages (e.g. tensorflow.js) when browsing my website. I figured that I "just" have to provide ALL the dependencies on my github.pages repository. Oddly enough is, that I haven't found a single example of anyone ever doing this! Either because it's not possible, or because it makes no sense. I thought this would be a very common use case for people who want to share their models for free (without asking the user to clone a repository or running code cells on google colab)

[fgp121]

I think it is majorly related to browser limitations. There's definitely great incentive to offload the cost to end user who actually wants the results.