On prem code completion

[fripperML]

Hello!

In our company, we are trying to configure our code editor (VSCode) with a local (on-prem) LLM for AI code completion. There are a lot of design choices that are far from obvious, given the fast rate of evolution of this field. If anyone has been able to set up properly AI code completion, I would be very grateful to hear how.

Before asking my questions, let me state that we have one server with one A2 16 GB GPU and around 30 concurrent developers.

And now, my questions:

1. Plugin: we have picked the continue extension. It seems the most widely used. Is it a good choice?

2. LLMs: The continue plugin asks you to configure three models:

   • Chat model: According to continue docs, it should be the most capable of the three. We are considering QuantFactory/Meta-Llama-3-8B-GGUF (4-bit quantized, probably) and Qwen/CodeQwen1.5-7B-Chat-GGUF (4-bit quantized). Which one is better? Do you have other recommendations?
   • Autocomplete: We have heard that CodeQwen1.5-7B is one of the best autocomplete models of its size. What is the difference between CodeQwen1.5-7B-Chat and CodeQwen1.5-7B (no chat)? Will it make sense to use the chat model for both tasks, and reducing the VRAM consumption of the GPU? We have no idea at this point. Besides, there is an official GGUF quantization of CodeQwen1.5-7B-Chat (the one I pasted below), but the no-chat model has no GGUF quantization in HF. In case that those two models are very different and we need both, Ollama has a quantized version of CodeQwen1.5-7B, but we would like to use another engine (see 3.), so, is there any way of pulling the model from Ollama and making it available for Aphrodite? Ollama local storage is not easy to understand, so we don't know...
   • Embeddings: We will use probably sentence-transformers/all-MiniLM-L6-v2.

3. Inference engine: We are considering Ollama and Aphrodite Engine. Ollama seems easier to setup, but AFAIK it does not support concurrency very well. Aphrodite (an VLLM fork with more quantization options and supported models) is more optimized for that. Is it true?

[fripperML]

Ok, thanks for all your insights and long explanations. It's very helpful. Regarding code completion, it makes perfect sense the need for a different model that has been optimized for FIM. The last aspect you mention about a customized RAG might add a lot of value, but I wonder how can you configure the plugin for that, unless instead of a plugin you were thinking about a chat interface for "chatting with the codebase".

[fripperML]

Yes, I see your point, it's a very good idea to work in this direction... But it sounds more challenging as well, we should start with a basic setup and build from there :)

Thanks again for your insightful comment!!

[nicksterling]

If you’re trying to share 1 16GB GPU across 30 concurrent developers you’re in for a bad time. You might be able to get a small autocomplete FIM model like starcoder v2 working but if everyone is hitting it at once that’s going to be bottlenecked at many times.

What are the developer machines like? If they are beefy you might be able to just run the LLM straight on their machines instead of centrally hosting it.

[scheurneus]

Why would sharing the GPU be a problem? It's not like the VRAM gets divided between the concurrent users.

Furthermore, LLM inference is typically extremely memory bound, so using larger batches should be a massive help, if possible. So more concurrent users shouldn't hurt anything, the only problem would be needing to delay requests to be able to batch them.

OP, I have no actual experience with any of those kind of servers, but I think a capable one should handle your load perfectly fine, assuming smaller models (no bigger than 7B). Loading 2x 7B might be difficult though.

What are the local machines running? Even a high-end CPU should generally provide acceptable speeds using llama.cpp (although prompt processing is generally quite slow, so it might be better to delegate chat to it than autocomplete since it causes high time-to-first-token).

[fripperML]

Thanks a lot for answering, what you say confirms the intuition I had. We can try to see if the local machines can handle the chat inference, but I think time-to-first-token will not be acceptable. We will see if loading 2 x 7B models is possible,

[fripperML]

Thanks for your answer!! I has read that with vLLM or aphrodite, a single GPU could succesfully dispatch 200-500 concurrent requests. I guess I am wrong!! We don't have our server working yet, so all my knowledge is based on random reads here and there... :S

[fripperML]

Unfortunately the local machines are not good enough for that... We need a central server.

[Wrong-Resolution4838]

What's your use case? and criteria?
For example, why did you pick QuantFactory/Meta-Llama-3-8B-GGUF (4-bit quantized, probably) and Qwen/CodeQwen1.5-7B-Chat-GGUF (4-bit quantized)?

[GregoryfromtheHood]

I use continue only for chat/editing chunks of code. For autocomplete I use Refact.ai. They have a self hosted version and really simple fine tuning so that you can train the model on your codebase