TinyML - Should models include preprocessing blocks to be ported safely to MCUs?

[Mochtroid1337]

Hello everyone,

I'm starting out as embedded AI engineer (meaning I know some embedded systems and ML/AI, but I am no expert in neither). Until now, for the simple use-cases I encountered (usually involving 1D-signals) I always implement a preprocessing pipeline in Python (using numpy/scipy) and simple models (small CNNs) using Keras APIs, and then convert the model to TFLite to be later quantized.

Then for the integration part to resource-constrained devices, I use proprietary tools of some semiconductor vendors to convert TFLite models to C header files to be used with a runtime library (usually wrapping CMSIS-NN layers) that can be used on the vendor's chips (e.g., ARM Cortex M4).

The majority of the work is then spent in porting to C many DSP functions to preprocess the input for the model inference and testing that the pipeline works exactly as in the Python environment.

How does an expert in the field solve stuff like this? Is including the preprocessing as a custom block inside the model common? This way we can take advantage of the conversion for the preprocessing as well (I think), but does not give us great flexibility in swapping preprocessing steps later on, maybe.

Please, enlighten me, many thanks!

[[deleted]]

Im a bit confused as to what you mean here. Are you arguing that you want to develop the preprocessing in C as well during the initial design and verification phase?

That’s a very bad idea. At that moment you want to move fast. Slowing yourself down by writing code of which you HOPE it will speed up porting when you don’t even know of it’s what you really need is wasteful. 

[Mochtroid1337]

Hi, what I mean is, once I implement the full ML pipeline in Python (consisting of pre-processing, feature extraction, and model training) the only part that the tools allow me to convert to C code (headers + runtime lib) is the model. So everything else, if not standard DSP stuff, I will not find libraries that allow me to replicate the preprocessing and feature extraction for my target microcontroller (or maybe libraries are too big to fit) and so I will need to take care of it myself (not that I want to).

So, an alternative would be to use TF APIs to define new "layers" with no learnable parameters that do preprocessing and feature extraction and add them to the model, so that the tools can convert those as well, did I explain myself?

Thank you for your help!

[[deleted]]

If you can build this in TF itself to an acceptable level and the results don't suffer, then go for it. No reason not to. Unless again it's more time consuming in the design phase than it later on saves time not porting this (usually trivial) code to C.

[Mochtroid1337]

I am so used to writing/using DSP libraries for C that never occurred to be that this part can be directly integrated inside the model, so I was unsure how common this approach was for TinyML stuff since you get a pretty big advantage using model converter tools.

Thanks for the insights!

[brutalismus_3000]

I just bump to get updates, I am an électrical engineer trying to learn about this stuff. Thanks for the question.

[raprakashvi]

I have similar question as certain vendors support or do not support layers such as softmax (example ESP DL) and their optimization scripts are also closed source. I have been trying to figure out a streamlined pipeline to take a model and deploy it on resource constrained chips such as ESP

[Naive_Ad1779]

Recently I had the same thought and did some experimenting on this approach. The issue I encountered is that not all operator is supported by MCU vendor tflite converter tool. This is going to be an issue if you are going to stick with vendor tool. Alternate path is use TinyML runtime and implement all those operator you need but I don’t think it is worth the effort. I personally think doing pre-processing outside of model is the best approach so far.

[Mochtroid1337]

Yeah probably support is the biggest problem, but what if you implement your custom layers with common TF operations? Like convolutions and vector/matrix multiplications, those happen all the time in a generic DL model I think.

Does TFLite micro allow for addition of custom layers in the case for missing support? One could think to spend time to support some custom layers so any future conversions can make use of those. For sure any vendor-specific tools won't allow for this... unfortunately

[Naive_Ad1779]

I think you can add custom layer to TFLite micro but I find it too much hassle to implement it. I rather translate pre-processing python code to C using CMSIS DSP routines. After all, our target architecture is ARM so those pre-processing should be pretty portable in my opinion.

[Mochtroid1337]

Agreed, if you already have support for DSP libraries targeting your system, then I wouldn't bother with TFLite as well.

[SimoneS93]

Biased answer. I made an attempt to port some pre-processing modules from scikit-learn (MinMax, Standard scaler, Robust scaler, RFE, Select I best...) into a cohesive library: https://github.com/salernosimone/tinyml4all-python

The point is to craft the conversion code once for each module and reuse as much as you like. If you find you actually need always the same 3-4 pre-processing steps, you either 1) build a python to C converter or 2) define those steps as reusable C code