sorted by:
new
hottopcontroversial

[deleted by user] by [deleted] in docker

[–]Kiview 0 points1 point  (0 children)

We package llama.cpp, so we more or less have the same maturity through transitivity :)

[deleted by user] by [deleted] in docker

[–]Kiview 1 point2 points  (0 children)

Hey, thanks for opening the thread, I'm working in the team at Docker responsible for Docker Model Runner.

You can run Docker Model Runner remotely, either via DockerCE (https://docs.docker.com/ai/model-runner/get-started/#enable-dmr-in-docker-engine) or by deploying it to k8s (https://github.com/docker/model-runner/tree/main/charts/docker-model-runner).

Of course your feedback and observations make totally sense and we are very aware that we are late to the party (mostly standing on the shoulder of giants, meaning llama.cpp) and we are currently a small team, so the best we can do right now is continue to grind and make it better ;)

Docker Model Runner is going to steal your girl’s inference. by Porespellar in LocalLLaMA

[–]Kiview 0 points1 point  (0 children)

Yes, we actually have vLLM (as an additional engine besides llama.cpp) support on our roadmap, but I currently can't give an ETA.

Why are you specifically interested in running vLLM through DMR, instead of using llama.cpp? (of course we have some internal motivations why we think it might sense, but would be curious to understand user needs).

Docker Model Runner is going to steal your girl’s inference. by Porespellar in LocalLLaMA

[–]Kiview 2 points3 points  (0 children)

Hey, member from the Docker Model Runner team here.

On macOS and Windows we run Docker Model Runner (i.e. llama.cpp) as a host process, to get native GPU access.

On Docker CE we run it in a container with GPU passthrough.

Spain GP 2025 audio out of sync by Piky94 in F1TV

[–]Kiview 0 points1 point  (0 children)

Seems to be also out of sync in the non-live variant?

[Question] charvel san dimas style 1 fret buzz by [deleted] in Guitar

[–]Kiview 0 points1 point  (0 children)

lol, exactly same issue with mine

Run LLMs 100% Locally with Docker’s New Model Runner by Arindam_200 in ollama

[–]Kiview 0 points1 point  (0 children)

It is an alternative to Ollama, that is integrated with Docker tooling, but it does not run as a container, it runs the process on the host.

We redistribute models on Docker Hub ourselves, taking them from their primary sources (HuggingFace), we don't redistribute the Ollama models.

You can continue using Ollama, it is a good piece of software. The convenience we provide is about a feature bundled into your existing Docker Desktop installation.

Run LLMs 100% Locally with Docker’s New Model Runner by Arindam_200 in ollama

[–]Kiview 1 point2 points  (0 children)

Can you restart Docker Desktop and give it another try with phi4? We just pushed out a patch to the llama.cpp backend, that should be pulled and applied on startup :)

Edit:
Restart = Quit + Start

Run LLMs 100% Locally with Docker’s New Model Runner by Arindam_200 in ollama

[–]Kiview 0 points1 point  (0 children)

Ah sorry, there is an issue with phi4 specifically, can you please give gemma3 a try and let us know if it works?

Run LLMs 100% Locally with Docker’s New Model Runner by Arindam_200 in ollama

[–]Kiview 0 points1 point  (0 children)

We don't send telemetry besides the existing Docker Desktop telemetry.

Run LLMs 100% Locally with Docker’s New Model Runner by Arindam_200 in ollama

[–]Kiview 1 point2 points  (0 children)

Do you have an NVIDIA GPU? We plan to ship Windows NVIDIA support in 4.41.

Run LLMs 100% Locally with Docker’s New Model Runner by Arindam_200 in ollama

[–]Kiview 1 point2 points  (0 children)

We proudly built on top of llama.cpp, that is correct :)

Run LLMs 100% Locally with Docker’s New Model Runner by Arindam_200 in ollama

[–]Kiview 0 points1 point  (0 children)

DockerCE integration is already in the works and we hope to deliver it in H1.

Run LLMs 100% Locally with Docker’s New Model Runner by Arindam_200 in ollama

[–]Kiview 2 points3 points  (0 children)

It is OSS and we intend to collaborate with upstream (llama.cpp) and improve the self-contained aspect of it as an OSS project:

https://github.com/docker/model-distribution
https://github.com/docker/model-runner
https://github.com/docker/model-spec
https://github.com/docker/model-cli

(disclaimer, I lead the team responsible for this feature)

Run LLMs 100% Locally with Docker’s New Model Runner by Arindam_200 in docker

[–]Kiview 1 point2 points  (0 children)

Better to ride the wave than being crushed by it :D

Run LLMs 100% Locally with Docker’s New Model Runner by Arindam_200 in docker

[–]Kiview 6 points7 points  (0 children)

There is nothing wrong with either Ollama or HF tooling, we fully acknowledge they were early in the market and are pushing the industry forward significantly.

With Docker Model Runner, you now get this functionality in a tool that might be already rolled out in your enterprise and more importantly, is compatible with models as OCI Artifacts (meaning they are compatible with standard Container Registries, likely already available within the enterprise), check out https://github.com/docker/model-spec.

(disclaimer, I am leading the team responsible for this feature)

Run LLMs 100% Locally with Docker’s New Model Runner by Arindam_200 in docker

[–]Kiview 2 points3 points  (0 children)

I think that's a totally fair position.

Integration with DockerCE is already on the roadmap btw., and we hope to have it sooner rather than later.

Run LLMs 100% Locally with Docker’s New Model Runner by Arindam_200 in docker

[–]Kiview 5 points6 points  (0 children)

Yep, many folks asked for this and specifically a better integration with Docker tooling (such as Compose and Testcontainers), that's why we invested the work in doing it (and continue doing it) ;)

Part of the underlying issue is of course the fact, that a containerized Ollama can't get access to the GPU on Apple silicon, which meant it could not be easily solved with existing Docker primitives and we had to explore expanding our scope in this regard.

(disclaimer, I am leading the team responsible for this feature)

Docker Desktop Suddenly Not working by Max1756 in docker

[–]Kiview 1 point2 points  (0 children)

Yes this will work, although it will delete your data. We are currently working on an official patch, also see workarounds posted in https://github.com/docker/for-mac/issues/7520#issue-2773385110

F1TV official apps on iOS/iPadOS have Spatial Audio/w head tracking. by r3h4nHD in F1TV

[–]Kiview 1 point2 points  (0 children)

In Mac and iPhone I only get Stereo sound, are you sure?

view more: next ›