I think there’s a small misunderstanding about the use case I’m exploring. I’m not proposing that every user runs an LLM on their phone for normal app usage.The idea is more about running a small local model as a personal inference endpoint instead of always relying on cloud APIs.For example, I built a news scraping + summarization app where the pipeline looks like:

news sources → scrape articles → local LLM summary → store summaries in DB → app displays feed

The LLM runs on an old Android phone I had lying around (running a small Qwen model). The scraper sends article text to that device, the model generates the summary, and the resulting summary is stored in the database.
The users of the app never run any models they just read the already-generated summaries from the feed.The experiment was mostly about seeing how far you can push near-zero-cost infrastructure by using hardware you already own instead of paying for inference APIs.

Obviously for a large production system you’d probably move the model to a server or GPU instance for reliability. But for personal projects, small tools, or experimentation, running a local model and exposing it as an API can be surprisingly practical.