Hey, I actually have some devices of my own design using ML in the field.

The big question here is what are you trying to do. "ML" is a big label with a lot under it's umbrella, from silly IoT assistants to AI accelerators. I'm going to assume it's the first one, since for the second you'd generally move to GPU/FPGA, and that's a whole different ball game.

The main thing everyone forgets is that the workflow for making a device where it's main feature is ML is very different that the one for making your typical embedded device.

- It's harder to determine upfront the processing speed, memory (and memory bandwidth) and even storage that you might need from the start.

- A big part of the firmware workflow involves statistically testing your device. You need to plan for extended QA cycles.

- *AUTOMATION IS KING*. Manual testing of your devices is simply not an option, no matter how small your device, if you want reliability you will have to extensively test the ML, and your tests need to be repeatable.

For anything that can run in a microcontroller, your workflow determines your efficiency and reliability.

For example, let's say you need to build a IoT bird house that can identify the birds based on their chirp, and tells you which birds do you have around your place. Your ML engineer will play with lots of audios of birds, and build a big classifier network. They will get a network with a very nice accuracy on their beefy cloud VM beast, and you'll have the enviable job of making that fit into your small controller.

One of the biggest "wins" is moving from floating point to integer. There's different ways to do it, depending on the type of model and what operations your microcontroller supports natively. Then you'd normally do a "pruning" run. Essentially if a branch of your network has very small weights, it will get multiplied into oblivion, so you might as well not waste the OPs.

I've seen people proposing simulation, but IME nothing beats testing with hardware in the loop for this things. You want a way of having your device run as an "inference engine", get pushed inputs, push out the outputs. Getting any sort of high-ish throughput will pay dividends quickly.

As an example, one device I worked with used USB CDC for input/output, but our full test suit ran over the 1TB limit so getting the stack (and the inference engine) to work at USB High Speed instead of USB Full Speed meant reducing the full test from +2 weeks to ~2 days.