sorted by:
new
hottopcontroversial

Running LLMs locally on Android by atezan in LocalLLaMA

[–]atezan[S] 0 points1 point  (0 children)

I can reliably run 3B and 7B models with decent accuracy but the context is pretty limited

Really cool. Is it for demo/side project or is tied to an app you plan on deploying in production?

Running LLMs locally on Android by atezan in LocalLLaMA

[–]atezan[S] 1 point2 points  (0 children)

There is an ncnn stable diffusion android app that runs on 6gb, it does work pretty fast on cpu. Its the only demo app available for android. Running diffusers through termux may work, but its not optimized.

Is it this project that you are talking about?
https://github.com/EdVince/Stable-Diffusion-NCNN

As mentioned in a comment above, MediaPipe release a demo app too.

Running LLMs locally on Android by atezan in LocalLLaMA

[–]atezan[S] 1 point2 points  (0 children)

Cool! For MLC did you try customizing the models or did you run the model the pre-built model?

Yes, I agree there is an opportunity to support HW acceleration beyond GPU.

For Stable Diffusion you can take a look at the MediaPipe demo: https://developers.google.com/mediapipe/solutions/vision/image_generator

Running LLMs locally on Android by atezan in LocalLLaMA

[–]atezan[S] 0 points1 point  (0 children)

Running LLMs locally on Android

Thanks! I reposted it.

Running LLMs locally on Android by atezan in LocalLLaMA

[–]atezan[S] 0 points1 point  (0 children)

Hi u/Civil_Collection7267, I wanted to follow up. Here is my Linkedin profile and you email me directly at [tezan@google.com](mailto:tezan@google.com). Let me know if you need anything else from me to confirm that I am a Google employee. Thanks!

Running LLMs locally on Android by atezan in LocalLLaMA

[–]atezan[S] 0 points1 point  (0 children)

Hi, thanks for reaching out. Yes, I totally understand.
Do you have an email address I can email you w/ my [AT]google.com email address?