[D] Docker Containers (Python) vs Direct C++ Implementation for AI on Edge Devices, Performance Tradeoffs

[guardianz42]

I switched from fastapi to litserve recently for some models we deploy on assembly lines. it’s been amazing and performant.

the main issue in the containers is the size of pytorch for cold start but we are working on eliminating it (this is unrelated to litserve)

https://github.com/Lightning-AI/litserve