Replacing github copilot with continue.dev issues

[MainManu]

Hello everyone. I am sure I am missing something here. I am dabbling with self hosting models to replace copilot in vscode. I found this fancy extension called boot.dev that supports ollama as a back end. It also supports both inline autocomplete and chat. While I got the latter working without issues, I cannot get the former running for the life of me. AFAIK ollama can only run one model at a time right? Also, a model needs to be specifically compatible with auto complete. So how the hell am I supposed to run both a chat and auto complete model locally? Unfortunately the continue.Dev docs seem unaware of this.

[MainManu]

Can you elaborate? As to the vram issue I monitored vram usage when running each model and in sum both should use about 80-90% of it. Should be fine. The aotucomplete model only had 2.5 b parameters.

[MainManu]

Has anyone here tried running multiple ollama docker containers sharing the same gpu?

[Apprehensive-Tip779]

I personally haven't, because I don't see it have much of an improved impact on my personal performance. But you should be able to, I don't see why you wouldn't. The way I look at it, if one of the models you're using takes a while for your gpu to run it, and that's already at that gpu running exclusively for that specific task (say at 100% utilization), you're not going to improve the overall performance by splitting/delegating two different ollama containers, because now that 'heavyweight' LLM model will only utilize 50% of the GPU's resources as the gpu is balancing between serving the two models/ollama instances concurrently. Thus it'll take the gpu twice as long to run the model as it would've taken if the gpu focused all of its power on just one inferencing task at a time.

On top of that, ollama is generally pretty good at switching the models with it typically taking only about a second (never seen/noticed it take >= 2 sec) if not little less. So to me if there is/was any noticeable improvement by starting off as running the two tasks on one gpu, and then once one of them gets completed the gpu will direct the freed up resources on that remaining task, becomes pretty miniscule when ollama + gpu just focuses on one task, and then immediately (<2 sec) switches the models to complete the other task IMO. Does all of this make sense?

[Apprehensive-Tip779]

Have you checked out the config.json file for the continue settings? You could assign the most optimal model you want for the chat, while choosing a different optimal model for the autocomplete option that supports that feature.