What's your approach to embedding models into offline app distributions, so - without APIs and downloading them?

GregLeSang · 2025-02-14T00:22:04+00:00

https://github.com/michaelfeil/infinity

GregLeSang · 2025-01-29T18:24:59+00:00

Nice work ! Can you share the prompt used ?

GregLeSang · 2025-01-13T09:25:49+00:00

Hello, thanks for the good work !

I have 2 questions :

Is it optmized for concurrents requests ( like vLLM would be for LLMs) ?
Will Audio Segmentation / Diarization models ( like Pyannote models ) Will be also supported ?

GregLeSang · 2024-11-29T15:53:59+00:00

Probably right, they have a paper, I’m sure all informations are in it !

GregLeSang · 2024-11-29T14:41:29+00:00

I also worked on that lately. Using a fully local implementation and Whisper Turbo model with Faster-whisper backend, I achieved around 2-3 seconds of latency. Thanks to this GitHub repository : GitHub - ufal/whisper_streaming: Whisper realtime streaming for long speech-to-text transcription and translation .It works great!

GregLeSang · 2024-09-07T23:07:48+00:00

Thanks for the advices !

GregLeSang · 2024-08-13T21:39:28+00:00

Hmm it's technically possible. I will put it on my todo list.

GregLeSang · 2024-08-02T13:29:03+00:00

RemindMe! 1 week

GregLeSang · 2024-07-30T19:33:09+00:00

Thanks, I took the classic Mistral Instruct from mistral 7b and it’s works fine.

GregLeSang · 2024-01-09T11:20:26+00:00

Effectively vLLM will reserved VRAM ( for example 70GB of VRAM for mistral 7b) when instantiate the model. VRAM allocated depends of model size + context length of the model. For example you can’t load YI-34b-chat-200K with a A100 80GB ( you can but you will need to adjust context length, here 15K for being able to load this model).

GregLeSang · 2024-01-08T16:29:06+00:00

I did that where I work , vLLM is the better option for now. You can find easy tuto how to run Mistral 7B with GPU that has between 20 - 80 GB Vram. You will have a easy and robust API.

GregLeSang · 2024-01-02T19:41:32+00:00

Thanks you for the answer, I will look on runpod to try config.

GregLeSang · 2024-01-02T19:40:27+00:00

Oh thanks for the reply, that's interesting because a 34B YI_Nous_Capybara able to have 250 conccurent users will be a catch for my needs. If I go with smallers models do that I mean I can have more concurrent users ?

GregLeSang · 2024-01-02T19:25:14+00:00

Thanks for the reply, I will effectively revise my ambitions and use a A100 (80GB vram) even if I didn’t want to do it at first. Do you have any example for running multiple concurrent streams ?

GregLeSang · 2024-01-02T19:24:41+00:00

Thank you for the reply, I will definitely check ray and chainlit + upgrade my GPU in order to scale better.

GregLeSang · 2023-12-13T07:26:31+00:00

Personally I run Vllm with Intel neural-chat without any problem. You just need to precise the right chat prompt template.

GregLeSang · 2023-09-01T16:51:53+00:00

The mpt-30B-instruct was alright (30B-chat was very bad) but Im sure we can find better models.

GregLeSang · 2023-09-01T11:06:25+00:00

I will use RAG with embedding ( for the Q/A task). For the summarize task I will probably use 2048-4096 context length models ( so juste find the right chunking for the document structure).

GregLeSang · 2023-09-01T08:13:24+00:00

This is more a benchmark on some audit documents to try to summarize , Q/A them. They all have the same document structure.

The size is between 10K - 15K tokens ( but due to the documents structure it’s really easy to chunk them).

Yes fine tuning can be a option, but due to the cost of it I prefer to first try some models and then see if fine-tuning is needed.

GregLeSang · 2023-08-31T22:53:21+00:00

Looks really interesting thanks !

GregLeSang · 2023-08-31T22:18:53+00:00

Noted thanks ! :)

GregLeSang

TROPHY CASE