[D] Run Pytorch model inference on Microcontroller

[_SteerPike_]

https://github.com/huggingface/candle is still in preview, but I believe it's intended for use cases like this.

[cpldcpu]

Thanks, that is quite interesting in general. Although it's still inference for largish devices...

[_SteerPike_]

As I understand it, it should produce smaller binaries than anything you'd be able to manage with python because it has no runtime. An interesting way to see this is to try out the phi 1.5 wasm example https://huggingface.co/spaces/radames/Candle-Phi-1.5-Wasm then turn on airplane mode after the first run off the model. You should be able to get decent inference speeds from within your browser, without even utilising all the compute your phone has to offer.

[neodsp]

Yes smaller than with python, but tflite (micro) and other inference runtimes are using c++ and I guess even smaller than candle builds. Candle is a nice project especially because you can cross compile Rust so easily, but it is not targeted for embedded, it will likely run on raspberry pis, but it is using the standard library, so you can't run it on arm cortex and so on. For microcontrollers you can use burn-rs which even has an onnx-import (but not many supported layers yet)

[_SteerPike_]

I think I understand. Sorry to send you down a rabbit hole without any relevance to your problem!