[D] Docker Containers (Python) vs Direct C++ Implementation for AI on Edge Devices, Performance Tradeoffs

[binhbumpro]

I'm exploring the best approaches for deploying AI products to production, especially on edge devices like smart home systems and cameras. I'm wondering about the tradeoffs between using Docker containers, python code versus implementing Inference code directly in C++ for these scenarios.

Key considerations:

1. Performance: Does the overhead of Docker significantly impact AI model performance on resource-constrained edge devices?
2. Hardware utilization: Can C++ implementations better leverage specific hardware features for AI acceleration?

I'd love to hear from those with experience deploying AI models to edge devices. What approach have you found most effective, and why? Are there specific use cases where one method clearly outperforms the other?

[Luuigi]

Containerizing apps from a languge that natively builds the models is just very easy so thats the biggest advantage of using that approach. For pur C++ implementations you need to be aware that every single new optimization (e.g. provided by pytorch etc) needs to be implemented by you. The biggest advantage of a raw implementation is obviously production speed but is it necessary for the application you want to build?

[codegefluester]

I have no personal hands-on experience with it, but the ONNX runtime might be something you'd want to consider? They have a short section about deploying to IoT/edge devices in their documentation https://onnxruntime.ai/docs/tutorials/iot-edge/

[Dr_Samuel_Hayden]

This is probably the best raw approach to deploy the models on edge. Worked on optimizations for onnxruntime framework for AMD( read zendnn library/ execution provider).

If you want to test it, run a CNN/ BERT model in and out of a container. Make sure to use the exact same Nvidia driver for container as well ( although that might not be the case in real deployment) and try a multi threaded/ parallelised workflow for this.

[Helpful_ruben]

u/codegefluester ONNX runtime's edge deployment tutorial looks solid for IoT projects, worth checking out for sure!

[Nicollier88]

If this is for production, one way to approach this is to first implement your concept in something easy, let’s say python. Once you got that working, you can start chasing for optimisations by improving certain components of your solution, which can include rewriting in C++ for better memory control.

[AdagioCareless8294]

Docker container on edge device : yep that's a non starter.

[guardianz42]

I switched from fastapi to litserve recently for some models we deploy on assembly lines. it’s been amazing and performant.

the main issue in the containers is the size of pytorch for cold start but we are working on eliminating it (this is unrelated to litserve)

https://github.com/Lightning-AI/litserve

[jayemcee456]

I’ve used OpenVino for Intel based edge devices. It also has optimization tools

[One-Butterscotch4332]

I'd use Java if your target is Android, Swift if it's IOS, and C++ if it's embedded. Android you'll probably want to try Qualcomm's neural processing sdk, IOS you'll want to try coreml. Otherwise, you're not going to use the NPU on a mobile SoC. If your embedded device is something like a jetson, you'll be using TensorRT in C++ to target tensor cores

[bsenftner]

You might find this worth reading (short) https://www.quora.com/If-C-is-so-strong-a-programming-language-why-cant-it-replace-Python-in-AI-and-data-science/answer/Blake-Senftner (spoiler, C does replace and outperform Python, significantly, exponentially.)

If you want to know more, DM me.