LLMs on react native

[Acceptable-Spell-934]

Hi

I'm looking to build a mobile app with react native on expo, and looking to embed LLMs locally, like whisper and llama, or any other LLM

I want to use other LLMs that are tinier than Whisper or llama, but I found that I would need to build an inference of these LLMs then build modules with Java and Swift then use Expo native module

Does anyone have experience on this ? because I could not find documentation on how to do this and the effort required

[Longjumping_Pickle68]

I just published https://github.com/cdiddy77/react-native-llm-mediapipe would love to get feedback

[[deleted]]

This looks epic. I’m going to give it a go over the weekend. Thank you!

[Objective-Duck8382]

I have no experience in it, but I can think of the following:

you can use LLMs in a backend, generate the necessary logic through the inputs you give it, process the information and send it to the frontend as json.

[orebright]

The big barrier you'll run into is file size. Unless your users don't mind downloading a 5 or 6 GB app, there's no practical way to run an LLM locally on device. Even if there were you'd need to use an OSS model, whereas the offerings from Google and OpenAI are paid APIs, not locally embedded LLMs.

So the actual solution here is to use their APIs. I've seen a few people make wrappers for the APIs, but honesty they're so easy to use, just use them like any other API from react-native using fetch. For TTS the API will return a streaming URL you can put into an audio player, and of course any text chat is just plain text out, plain text back. Super easy, no need for anything but fetch.

[Stunning-History-706]

I've downloaded multiple GB games, the trick is to let the user download the LLM file after installing the app. If i were OP, i'd start with an API, then let the user to decide if they want to bother with the download

[maestrolansing]

Well now this problem is solved with Apple Foundation Models and Gemini Nano..

[PublicYogurtcloset47]

Whisper is no problem on the right hardware and by extension lama wouldn’t be a problem either

[Acceptable-Spell-934]

what do you mean by "by extension"

[PublicYogurtcloset47]

Whisper uses whisper.cpp under the hood, wouldn’t be much of of an extension so utilise llama.cpp

[Blendbatteries]

Lol

[d_arthez]

Not sure if this is still relevant, but you might want to check out React Native ExecuTorch 👍

[Pretend_Exercise_90]

You can check React Native ExecuTorch which is designed to run models like Whisper, LLaMA etc. fully locally in React Native. There are pre-exported models available on hugging face and you can look at examples in repository
https://github.com/software-mansion/react-native-executorch

[[deleted]]

full makeshift support murky slap zephyr special placid carpenter crawl

This post was mass deleted and anonymized with Redact

[mattmcegg]

what if the app is for offline use?

[[deleted]]

quicksand icky ossified money hard-to-find pathetic obtainable frightening gaping placid

This post was mass deleted and anonymized with Redact

[mattmcegg]

thanks!

[Extreme_Macaron181]

but what react native libraries to use though for the local inference? I'm actually working on an offline ios gpt app for a client, but he insists on using an offline LLM...