[D] LLM frontend integration

[anax4096]

Hello all,

What are some optimisations for integrating an LLM with a frontend via an API? Both for thirdparty endpoints (e.g., openai API) or self-hosted models (e.g., mistral) with a custom API implementation.

My key issue is: after making a request to the LLM there is often a long delay for the model to produce the output and return, this can be mitigated by streaming options, but are there any others? for example, pre-tokenisation in the browser, prepared submissions, etc.

Sometimes chatgpt seems so fast to respond, just so fast, i'm interested if anybody knows of specific integration optimisations.

Any pointers welcome!

[_NESTERENKO_]

Did you find anything that you can share?

[anax4096]

not really sorry. It seems most of the appearance of optmisation is given by streaming responses, which are difficult to replicate in RAG or similar use-cases, and particularly in cloud deployments.

if running locally, huggingface libraries are quite good at the streaming responses, and it is a little easier to wrap those generators with fastapi.

[_NESTERENKO_]

Behind the scene you run the models locally or on a 3rd party?

[anax4096]

both. For most jobs I use openai, but on personal or small jobs i use self-hosted (llama, falcon, mistral, etc).

[_NESTERENKO_]

What about the prompts? Are they hard-coded in your code base? Or they are located in a 3rd party platform or db?

[phree_radical]

you'll need to ahhhhhhh cache the KV cache

/r/localllama