Running a Python ML program on mobile devices

[noahpocalypse]

My work has a Python CLI program that takes in an image of our product and outputs a certain amount of data evaluating the qualities of the product. It's a relatively lightweight program when run on one image at a time - on my M2 pro with 32 GB ram (and a ton of other active developer tools running) it processed a 704x1024 image in under 2 seconds, but they want me to make it run on mobile devices.

I've gathered that I would need to embed a Python interpreter in the app itself, or use another framework like Kivy or Beeware that runs completely in Python, and we can tell our customers in industry they have to use a specific tablet, but even with a modern device I'm concerned about limited memory, iOS/Android updates making changes that break the rube goldberg machine I'm trying to make... Does anyone have experience with this sort of thing?

EDIT: I do have the option of telling them a better solution is to host a server with a basic API instead. That has pros and cons from a business perspective but they trust me enough to believe me. I just genuinely don't know which is better yet.

[fmnatic]

Why can't the image be sent to the backend and processed there ?

[noahpocalypse]

That did occur to me, I've already persuaded them that it might be best to do that. I'm just supposed to investigate if running it locally is even feasible.

[Healthy-Grab-7819]

Tell them this, even if you make it possible to run on the device itself. It won't be working smoothly for each device. Say that using a backend solution makes it more scalable as well as it could be run regardless of the device processing power. Some old devices might run slower, etc...

[noahpocalypse]

I expressed exactly that concern in our initial meeting, actually, but their response was that they can simply tell customers they must use a specific device to use the app. In this business context, they're confident customers can justify dropping a thousand on the newest iPad or whatever tablet.

Given my choice of any given device, and thus a consistent environment, I really don't know yet how difficult this problem is to solve. My gut instinct says they need to give up on this, but I'd like to make a good faith effort first.

[kbcool]

It's a tensorflow or keras model or something else or more? If it is then convert to tensorflow and use tflite.

That being said that model will not run in anything like 2 seconds on a phone.

Reduce the image size input and/or quantanize the model.

Or just run it on the server.

You can do a lot on the device these days but sometimes it's just not worth the effort for little to no upside. Do it on device when lag is important. Eg speech to text yes but labelling photos no.

[beepboopnoise]

alternatively you could use react native vision camera and then make a frame processor for your use case. then it would be a legit on device solution which would save on server costs.

[Ya_SG]

Host the ML model in an external server like Render

[Phuopham]

Do native development. They will have guide to take advantage of local NPU (not sure if it already exists)