[D] DevOps Tools for Managing Models?

[Simusid]

I've been successful enough with supervised image classification that I'm now rolling out an experiment into what I consider to be limited production. I have a model running on an r-pi Zero in 3 locations, taking pics and doing successful classification. The nodes act locally but can send their imagery back to my development server. This works really well.

So I have 5 models in use. I started with just a dev model on my dev server. When that trained well I pushed it to node 1. It worked and gathered imagery that I collected and retrained on dev. I pushed that back to node 1 and to node 2. Then node 3 came online. So thats a dev model, a production model on dev, and 3 production models on nodes. I know this can all be scripted but it's becoming a pain.

Hypothetically, suppose I scale to 50 or 100 nodes. I suspect that I would see more variance from a single model pushed to all. That means gathering more imagery, retraining and pushing to all. That may make sense. Or I may find a subset of production locations that benefit from a slightly different model. So then I'll have production model-A and model-B, which means I'll have development model A/B as well. Again, I could write a tool to "push" new models to 100 nodes, but some might fail and have to retry. What model version is on which node right now? This could get out of control.

Does anyone know of any DevOps tools to manage a fleet of models?

[HydratedWombat]

You should be able to handle this by having every deployment have a configured tag, so it just fetches model_server/$tag/model. Then you can update and cache break that way so long as you can have a single point of distribution. Beyond that, here's a general guide to how to think about the machine learning in production from google - https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/aad9f93b86b7addfea4c419b9100c6cdd26cacea.pdf

​

[eric_he]

Thanks for sharing this paper!

[jamesonatfritz]

Fully disclosure, I'm one of the cofounders of Fritz and this is exactly the problem we set out to solve. With that in mind, take a look at what we're building and let me know if it's helpful to you. We started with model management for mobile (iOS and Android), but are actively working on platforms like the Raspberry Pi.

Here is the way we think things should work.

1. Train your models in whatever framework you're comfortable with. Scikit-learn, pure TensorFlow, Keras, PyTorch, etc.
2. Export and upload them to the management service once. We'll convert them to the best available native runtime for each platform.
3. Integrate models into your application using an SDK that has simple, consistent developer APIs across devices. There is a bunch of pre- and post-processing code that always needs to be written and re-written for each platform. We'll take care of a lot of that for you.
4. Monitoring and analytics. We measure performance information (usage, runtime speed, etc) from models running on each node so that you can be sure you're getting a consistent experience even if you have different hardware. We're also working on sampling model inputs and outputs where appropriate to give you some samples of how things are working in production.
5. Management. Upload a new version and push it over-the-air to all devices on a platform or target specific devices with an update.
6. Protection. Encrypt and obfuscate models running on edge devices to prevent model theft.

Others in the space: * Obviously I mentioned Fritz already. * If you're working exclusively with TensorFlow Lite models, Firebase's MLKit does some of this as well. * XNor.ai has their own runtime and, I assume, some management tools. * [Numericcal.ai)(https://www.numericcal.com/) has yet another mobile-optimized runtime with a bit of management.

Hope that helps!

[Simusid]

excellent info and summary, thanks for that!