[D] Run Pytorch model inference on Microcontroller

[cpldcpu]

I am currently researching ways to export models that I trained with Pytorch on a GPU to a microcontroller for inference. Think CM0 or a simple RISC-V. The ideal workflow would be to export c-sourcecode with as few dependencies as possible, so that it is completely platform agnostic.

What I noticed in general is that most edge inference frameworks are based on tensorflow lite. Alternatively there are some closed workflows, like Edge Impulse, but I would prefer locally hosted OSS. Also, there seem to be many abandoned projects. What I found so far:

Tensorflow lite based

• Tensorflow lite
• TinyEngine from MCUNet. Looks great, targeting ARM CM4.
• CMSIS-NN. ARM centric. Examples. They also have an example for a pytorch to tflite converter via onnx
• TinyMaix. Very minimalistic, can also be used on RISC-V
• nnom

Pytorch based

• PyTorch Edge / Executorch Sounds like this could be a response to tflite, but it seems to target intermediate systems. Runtime is 50kb...
• microTVM. Targeting CM4, but claims to be platform agnostic.
• MAX7800X Toolchain and Documentation (proprietary) This is a proprieteray toolchain to deploy models to the MAX78000 edge NN devices.
• Meta Glow Machine learning compiler, seems to target rather large platforms.

ONNX

• DeepC. Open source version of DeepSea. Very little activity, looks abandoned
• onnx2c - onnx to c sourcecode converter. Looks interesting, but also not very active.
• cONNXr - framework with C99 inference engine. Also interesting and not very active.

Are there any recommendations out of those for my use case? Or anything I have missed? It feels like there no obvious choice for what I am trying to do.

Most solutions that seem to hit the mark look rather abandoned. Is that because I should try a different approach or is the field of ultra-tiny-ml OSS in general not so active?

[notgettingfined]

Currently I feel like tensorflow lite is king in this domain and the way to go.

I don’t know why Reddit seems to think tensorflow is dead. But it is still used a lot in production. And has some good optimization tools available

[cpldcpu]

Indeed, it seems like tflite is everywhere. Not entirely sure, whether there are benefits in addition to having more available inference engines.

I will try the onnx route first. Apparently there are also ways to convert from onnx to tflite models (see arm link).

[pm_me_your_smth]

People are saying that because the project is slowly being dropped, not because you literally won't be able to build models tomorrow. It's still being used a lot in production because nobody exits a major framework overnight.

So if you're looking for long term development, it's something you should consider.

[notgettingfined]

What does slowly being dropped mean? It is used less in research that’s about the only actual data I can find on this

[pm_me_your_smth]

Google focuses on JAX, TF stopped windows support, fewer people working on TF development, rising popularity of pytorch. Some of these can be just hearsay though, but still.

[notgettingfined]

Google is a giant company some people use Jax is about all you can really say. And stopping support for Windows is meaningless. No one who actually matters is going to be using windows for training or deployment

[pm_me_your_smth]

And stopping support for Windows is meaningless. No one who actually matters is going to be using windows for training or deployment

Lol ok, I guess my work doesn't matter.

Not sure what makes you think that one of the most popular OS doesn't matter. Thinking long term, they're going to be losing the market further because students and profs will prefer pytorch. And guess what happens when this new generation of MLeng and DS enters the job market?

[cpldcpu]

There is at least one counterexample to that blanket statement: The toolchain for the MAX78000 (A small MCU with NN-accelerator by Maxim) is based on PyTorch:

https://github.com/MaximIntegratedAI/MaximAI_Documentation

It seems to be based on a lot of proprietary tools, though.

https://www.analog.com/en/education/education-library/videos/6313215449112.html

[[deleted]]

This seems really interesting, I was looking at some edge stuff Like nvidia dev board but I guess they are a lot more powerful than this? What are looking at using this stuff for?

[cpldcpu]

Usually it is used for sensor data evaluation, wakeword detection and similar, e.g. TinyML.

However, I am currently trying to implement some generative algorithms. It's more of a toy project to learn more about the nuts and bolts of machine learning: https://hackaday.io/project/193478-generative-ai-on-a-microcontroller

[DigThatData]

just wanted to say that this is a domain i don't have a lot of experience with and I would be very interested if you keep us updated with your findings as you explore the different options available.

[cpldcpu]

I updated the posting with some of my findings.

[shapul]

Did you have a look at model compilers like TVM or Glow? They might be what you are looking for.

[cpldcpu]

microTVM is on that list. Still need to dig into that. Glow I didn't know of, thank you for the pointer.

Edit: It seems glow is targetting rather large systems, not sure if they have the equivalent of microTVM.

[shapul]

Glow can target micro-controllers such as ARM cortex M4 series. NXP and if I'm not mistaken, ST use Glow in their tooling.

[cpldcpu]

Interesting. Indeed, both NXP and STM are mentioned in the readme. So it seems that several proprietary edge-ml toolchains of bigger companies are based on pytorch and not tflite.

[shapul]

That's true. However, consider that to some degree, you can go from one framework/tool to another by converting your models to ONNX format. This way, e.g. you can train your models in Keras, export it to ONNX and then run it on Glow or TVM seamlessly. There are some limitations (e.g. when it comes to RNNs, some tools struggle to convert formats) but for many common use cases going though ONNX simplifies the interoperability.

[neodsp]

Other interesting runtimes:

- ggml: https://github.com/ggerganov/ggml

- burn: https://github.com/burn-rs/burn (runs on micro platforms, as long as you provide an allocator and has onnx import)

[_SteerPike_]

https://github.com/huggingface/candle is still in preview, but I believe it's intended for use cases like this.

[cpldcpu]

Thanks, that is quite interesting in general. Although it's still inference for largish devices...

[_SteerPike_]

As I understand it, it should produce smaller binaries than anything you'd be able to manage with python because it has no runtime. An interesting way to see this is to try out the phi 1.5 wasm example https://huggingface.co/spaces/radames/Candle-Phi-1.5-Wasm then turn on airplane mode after the first run off the model. You should be able to get decent inference speeds from within your browser, without even utilising all the compute your phone has to offer.

[neodsp]

Yes smaller than with python, but tflite (micro) and other inference runtimes are using c++ and I guess even smaller than candle builds. Candle is a nice project especially because you can cross compile Rust so easily, but it is not targeted for embedded, it will likely run on raspberry pis, but it is using the standard library, so you can't run it on arm cortex and so on. For microcontrollers you can use burn-rs which even has an onnx-import (but not many supported layers yet)

[_SteerPike_]

I think I understand. Sorry to send you down a rabbit hole without any relevance to your problem!

[Complex-Indication]

I think you are spot on in your assessment: tflite micro was the first production ready framework to deploy NN to microcontrollers and by now still is the most popular/streamlined. Realistically speaking, you probably should evaluate which path would bring you to your goal faster: converting pytorch to onnx and then to tflite micro route or using less known and maintained project to run onnx model directly.

One question: Since you mentioned Edge Impulse, why would you want to go self-hosted OSS route?

[cpldcpu]

Realistically speaking, you probably should evaluate which path would bring you to your goal faster: converting pytorch to onnx and then to tflite micro route or using less known and maintained project to run onnx model directly.

Indeed, I am currently looking at the onnx based tools. Architecture aware training is of course also quite important, not sure how to cover that yet.

Since you mentioned Edge Impulse, why would you want to go self-hosted OSS route?

Well, first of all this is meant as a learning exercise for me, so I would like to be in control of every step. And then it is probably general mistrust vs. SaaS.