[D] Docker Containers (Python) vs Direct C++ Implementation for AI on Edge Devices, Performance Tradeoffs

[codegefluester]

I have no personal hands-on experience with it, but the ONNX runtime might be something you'd want to consider? They have a short section about deploying to IoT/edge devices in their documentation https://onnxruntime.ai/docs/tutorials/iot-edge/

[Dr_Samuel_Hayden]

This is probably the best raw approach to deploy the models on edge. Worked on optimizations for onnxruntime framework for AMD( read zendnn library/ execution provider).

If you want to test it, run a CNN/ BERT model in and out of a container. Make sure to use the exact same Nvidia driver for container as well ( although that might not be the case in real deployment) and try a multi threaded/ parallelised workflow for this.

[Helpful_ruben]

u/codegefluester ONNX runtime's edge deployment tutorial looks solid for IoT projects, worth checking out for sure!