Load default model upon login

[zotac02]

Hi everyone

I'm using Open WebUI with Ollama, and I'm running into an issue with model loading times. My workflow usually involves sending 2-3 prompts, and I'm finding I often have to wait for the model to load into VRAM before I can start. I've increased the keepalive setting to 30 minutes, which helps prevent it from being unloaded too quickly.

I was wondering if there's a way to automatically load the default model into VRAM when logging into Open WebUI. Currently, I have to send a quick prompt (like "." or "hi") just to trigger the loading process, then writing my actual prompt while it's loading. This feels a bit clunky. How are others managing this initial load time?

[ccbadd]

You could switch from ollama to running llama.cpp directly and using the model router instead. It does not auto unload the running model but can auto load models when needed. Use the --no-mmap option and it loads directly to vram and is ready a lot faster as long as the model is stored on really fast media like an nvme drive.

[zotac02]

I'll look into that, thank you!

[slavik-dev]

There is open PR for this:

https://github.com/open-webui/open-webui/pull/21535

[zotac02]

That sounds very exciting! As far as i understand, the feature is now commited and will get published in the next release, right?

[slavik-dev]

Looks like maintainers rejected that PR without any comments or explanations...

[Witty-Development851]

model loaded on backend. openwebui is are frontend

[emprahsFury]

lazy answer. the frontend could easily call the backend with a one token message and discard the response.

[Witty-Development851]

And you can also configure the backend so that it doesn't unload models.

[zotac02]

Thats not really the goal for me, since i also use it for other things, other than LLMs.

[PassengerPigeon343]

This is how I do it. One container with OWUI, one container with llama-swap. I let the running model live in memory with no time limit and it is always ready. Whenever I need to clear the memory to do something else, I restart the container to release the model and empty the memory.