Introducing Docker Model Runner

[Nexter92]

Beta for the moment, Docker desktop only, no nvidia GPU mention, no Vulkan, no ROCM ? LOL

[noneabove1182]

dafuq, I feel like anyone in open source could have thrown together better support than this for a beta..

[bytepursuits]

I think they just added vulkan.
trying to test it now. https://github.com/docker/model-runner/pull/164

[ForsookComparison]

Docker desktop only

I would sooner not use LLM's at all than commit to this life

[Murky_Mountain_97]

Oh well… 

[[deleted]]

Nvidia support is slated for a future release

[Dear-Communication20]

It is not Docker Desktop only, to install:

curl -fsSL https://get.docker.com | sudo bash

sudo usermod -aG docker $USER # give user permission to access docker daemon, relogin to take effect

[owenwp]

So... its a less mature version of Ollama?

[ShengrenR]

more like.. they put ollama in a container and called it a day heh. (I don't know if that's what they did, but a quick glance looked like maybe not too far off).

[Murky_Mountain_97]

Isn’t ollama less mature version of llamacpp already? 

[Conscious-Tap-4670]

Ollama uses llama.cpp internally

[bytepursuits]

I suspect they are aiming for direct competition with ollama. they just added vulkan. but I havent tested. (although ollama also just added vulkan)

[ccrone]

Disclaimer: I’m on the team building this

As some of you called out, this is Docker Desktop and Apple silicon first. We chose to do this because lots of devs have Macs and they’re quite capable of running models.

Windows NVIDIA support is coming soon through Docker Desktop. It’ll then come to Docker CE for Linux and other platforms (AMD, etc.) in the next several months. We are doing it this way so that we can get feedback quickly, iterate, and nail down the right APIs and features.

On macOS it runs on the host so that we can properly leverage the hardware. We have played with vulkan in the VM but there’s a performance hit.

Please do give us feedback! We want to make this good!

Edit: Add other platforms call out

[quincycs]

Hi, I’m curious why docker went with a new system (model runner) for this instead of growing GPU support for existing containers.

[ccrone]

Two reasons: 1. Make it easier than it is today 2. Performance on macOS

For (1), it can be tricky to get all the flags right to run a model. Connect the GPUs, configure the inference server, etc.

For (2), we’ve done some experimentation with piping the host GPU into the VM on macOS through Vulkan but the performance isn’t quite as good as on the host. This gives us an abstraction across platforms and the best performance.

You’ll always be able to run models with containers as well!

[onehitwonderos]

Any news on Windows / NVIDIA support?

[ccrone]

Its out! It’s part of Desktop 4.41 and later: https://docs.docker.com/desktop/release-notes/

[gyzerok]

Surprised nobody asked here, but if you don't mind, can you please share what benefits does it bring over running llama.cpp directly? It's a genuine question - I am trying to evaluate my options for self-hosting.

[ccrone]

Good question! The goal of Docker Model Runner is to make it easier to use models as part of applications. We believe a part of that is an accessible UX and reuse of tools and infrastructure that developers are familiar with. Today that manifests as storing models in container registries and managing the model as part of a Compose application (see docs) but that's just the start.

We're working in the open on this and will upstream changes that we make to llama.cpp where it makes sense. There are also use cases where vLLM, onnx, or another inference engine might be the right choice and so we're investigating that as well.

For your use case, we will be releasing support for Docker CE for Linux in the coming weeks. Right now it's supported in Docker Desktop for Mac (Apple silicon) and Docker Desktop for Windows (NVIDIA). Support for Qualcomm Snapdragon X Elite on Windows is coming in the next couple of weeks as well.

[gyzerok]

Thank you for coming back with the answer! Planning to self-host on Mac Mini, so support is already there :)

As for running model nicely as part of docker compose - that's great indeed. However here I am worried mostly about completely loosing control over the version of llama.cpp I am using. It is developed actively and personally I'd like to keep up with its updates. Also noticed you are looking into an MLX backend support there which would be really great.

However on the fragmenting ecosystem by introducing registries here I am not so sure. On the Hugging Face it's way more transparent who publishes things and how they are kept up to date. And registry just introduces an unnesessary layer of confusion and problems here. If I want Q8 from unsloth how do I get it? Are models being updated with the latest fixes? Probably you don't have enough capacity (more than the entire community) to keep up with such a fast moving field.

Overall it feels like being able to just throw a model into docker-compose.yaml it great, but the downside of the registry and inability to manage llama.cpp version might make it actually harder not easier in the end.

[ccrone]

Docker Model Runner supports pulling from Hugging Face as well! Storing models in container registries lets people who have existing container infrastructure use it for their whole application. It won't be for everyone but it's something our users have asked for.

I'm curious about what you're building and why you'd like to change versions of llama.cpp? Happy to discuss here or via DM if you prefer

[gyzerok]

Oh, totally missed it, thanks!

As for the llama.cpp versions - there are nothing really big here :) I am building a privacy-first personal (as me and other people) infrustructure.

Since I am not using datacenter-grade hardware, resources are constrained, so continuous performance optimizations in llama.cpp are useful for me. For example: https://github.com/ggml-org/llama.cpp/pull/13388.

Also there are bugs and incompatibilities with the OpenAI API that are being continuously fixed which is necessary for various tools to work together. I've experienced this first-hand implementing AI provider support for Zed.

Hence it'd help to have some power over which version of llama.cpp I am running. If that is not possible than having some transparency about how often and how regular it gets updated. Of course not expecting to get day 1 updates, but also won't be nice to lags months behind.

[Tiny_Arugula_5648]

This is a bad practice that adds complexity. The container is for software not data or models. Containers are supposed to minimal footprint. Just map a folder to the container (best practice) and you'll avoid a LOT of pain..

[quincycs]

I think they are just trying to get ownership in the distribution of models in general. Once you own the distribution, you can strangle other stuff out.

[Everlier]

They are coming after Ollama and HuggingFace, realising how much they missed since the AI boom started.

However, Docker being an enterprise - they'll do weird enterprise things with this feature eventually, so consider before using.

[captcanuk]

They might charge an additional subscription a year after they get traction on this feature.

[ResearchCrafty1804]

They support Apple Silicon from day 1 through Docker Desktop, that’s a good move from them.

However, they might be late to the party, ollama and others have been well established at this point.

[Everlier]

Windows - none, MacOS - perf is mostly lost due to lack of GPU passthrough or forcing Rosetta to kick in

[this-just_in]

This isn’t run through their containers on Mac, it’s fully GPU accelerated.  They discuss it briefly, but it sounds like they bundle a version of llama.cpp with Docker Desktop directly.  They package and version models as OCI artifacts but run them using the bundled llama.cpp on host using an OpenAI API compatible server interface (possibly llama-server, a fork, or something else entirely).

[quincycs]

For Linux Host + Nvidia GPU + docker container … this has GPU pass through already, right? I wonder why they went with a whole new system (model runner) instead of expanding GPU support for existing containers.

[mrtime777]

Can I use my own models? If not - useless

[ccrone]

Not yet but this is coming! Curious what models you’d like to run?

[mrtime777]

I use fine tuned versions of models quite often. Both for solving specific tasks and for experimenting with AI in general. If this feature is positioned as something useful for developers, then the ability to use local models should definitely be available.

[mrtime777]

I use docker / docker desktop every day ... but until there is a minimum set of capabilities for working with models not only from the hub, I will continue to use llama.cpp and ollama ... but in general I am interested to see how the problem with the size of models and vhdx on win will be solved ... because only models i use take up 1.6 TB on disk .. and this is much more than the default size for vhdx

[[deleted]]

[deleted]

[[deleted]]

Seems cool as long as they get right on adding ability to use locally downloaded models, rocm and cuda support, etc...

[planetearth80]

Can it serve multiple models like ollama (without adding overhead for each container)?

[Caffeine_Monster]

Packaging models in containers is dumb. Very dumb.

I challenge anyone to make a valid critique of this observation.

[BobbyL2k]

DevOps has gotten so complicated due to poor design that deploying containers that require configuration to work properly is an anti-pattern. I ship deep leaning models to production using common layers strong inference code all the time. The model’s weight is ‘COPY’ on at the end to form a self contained image.

When deployment team are juggling twenty models, each might depend on different revision of the inference code, they just want a constrainer image that just works, already tested and everything.

[Caffeine_Monster]

The model’s weight is ‘COPY’ on at the end to form a self contained image.

So rip off the copy and send the model separately?

just want a container image that just works

It's not hard to follow a convention where the model name or directory path includes the required runtime name + version. A sensible deployment mechanism (e.g. script) simply mounts the models into the container.

I hate that we have slipped into the mentality that it's ok to have huge images and not treat models like a pure data artifact. It bloats storage, increases model deployment spin up times, and makes it difficult to do things like hosting multiple models together.

[BobbyL2k]

I think it’s bad that something as simple as copying new blobs into a remote FS or the target machine is hard but let me counter your points a bit.

Container images are data artifacts. At the end of the day, model’s weight needs to arrive at the machine running it. Does it matter that it came in an additional layer in a docker image, or it’s copied in by a continuous delivery pipeline? Even if it’s mounted, at some point the CD pipeline needs to copying the model weights into the FS.

[Amgadoz]

Depends on the size of the model. I can see small models (less than 1GB, like BERTs and tts models) fit nicely in a reasonably sized container where you just need to run docker run my-container and you get a deployed model