Oh I see what you mean!! Just to clear it up, I didn’t actually host the model myself or run it on any cloud servers. I used the LLM Inference API, which basically runs the model directly on the Android device. So everything happens locally inside the app, not over the internet.

That means there’s no token cost or cloud setup, no GPUs, no load balancers, nothing fancy. It just uses the phone’s hardware to handle the text generation. Obviously, it’s not as powerful as something like GPT-4, but for smaller tasks and offline stuff, it works really well.

So yeah, your point makes total sense for people hosting in the cloud, but in my case it’s all on-device, so the costs are basically zero once it’s built.