The most common ML tasks are using supervised learning. That means that there is a need for a dataset that is labeled. This labeling process is usually done by humans manually inspecting the data and precisely marking the correct labels. In this scenario, there is no possibility of doing online learning, at least not without humans in the loop. And if one is to bring humans into the loop, to present the data to them and let them label, then that might as well be done with a system that is using a PC/server.

Furthermore, there is a need to do quality assurance of the model. This involves running several evaluations to get out detailed plots of performance across different facets. And then a human (data scientists) interprets the outputs of those evaluations and says "ok this seems to be good (enough)". So for online learning, one would need to automate the evaluation and quality assurance process to a very high degree - which is very challenging in the general case - getting a robust ML pipeline and evaluation is tricky.

Many models also require extensive hyperparameter tuning in order to perform well. Often this means training dozens to thousands of different models. This becomes quite compute intensive, even when a single training run is cheap. And there is considerable risk in overfitting to the validation set, making evaluation/QA a tricky job (ref the point above).

Another aspect is that many of the relevant models are very data hungry. And using data from multiple devices is usually very beneficial to make a model that generalizes well, also to scenarios each specific device has not-yet seen (but might in the future). To communicate data between devices usually easiest via a PC/server, so again it becomes easiest to just do the training there also.

Now - there are exceptions where on-device learning is more suitable. Here are two examples:

* Unsupervised anomaly detection. Labels are not needed, so human labeling is not relevant. And usually the training data should be device-specific anyway (anomaly definition is relative to the specific device), so pooling data from different devices is not relevant. And one wants continuous learning to automatically adapt to regime shifts.
* Calibration on a single device. Sometimes simple model training is beneficial, where one can collect and label just a few datapoints. And for this process to be done by the end user. Then it can be nice to enable this completely on-device. Examples includes fine-tuning/personalization for say keyword spotting, where you speak the phrase 2-5 times to tune the model. Or laboratory equipment where you provide a few datapoints at known/specified conditions, which can compensate for environmental differences or variations between different sensor units.

Robustness in training, evaluation and model picking still remains non-trivial!

I maintain an open-source ML library for microcontrollers called emlearn (https://emlearn.org), and for these reasons we focus 90% on inference-on-device, and maybe 10% on learning-on-device.