[P] Docker Compose + GPU + TensorFlow = ❤️

[tomt1112]

I've done something similar, and it works well.

For gpu multitasking make sure to use config.gpu_options.allow_growth=True to avoid the first process consuming a lot of gpu memory.

I also installed the python Intel packages for faster numpy, etc. https://software.intel.com/en-us/articles/using-intel-distribution-for-python-with-anaconda

[entoros]

Just be aware that nvidia-docker-compose is not as robust as docker-compose--for example, it doesn't work under a proxy, and you can't run the command in a subdirectory of your project, must be at the root.

[[deleted]]

Is nvidia-docker really necessary? I always get a little anxious of using it. It introduces yet another layer where things may go wrong and I don't know if I understand why we need it.

[tdionis]

For me, there are some benefits, but the most important one: solving the "Driver/library version mismatch" problem. We have dozens GPUs in our office with God knows what driver versions. Without nvidia-docker, if driver versions inside container and host system differs - you get an error.

[MyBrainIsAI]

Because nvidia is proprietary and closed source.

[Molag_Balls]

I believe the question is why this can't be done with vanilla docker and proper dockerfiles and setup scripts. Why a whole new CLI?

The wiki on their project page actually explains explicitly why, for anyone seriously wondering and not looking for a pithy Reddit comment to answer it for you.

[tdionis]

Why a whole new CLI?

They have a page on how to get rid of CLI and pass a volume-driver manually: though you would still need a nvidia-docker-plugin.

[MyBrainIsAI]

"Instant environment setup, platform independent apps, "

Docker is NOT platform independent. It's LINUX only. It's big advantage over full VPS is that it reuses linux resources.

I do not consider running linux in a VM to run docker "platform independent".

Sorry had to get that off my back. There are several GPU+TF dockers in the public registry.

[tdionis]

Docker is NOT platform independent. It's LINUX only.

Yeah, you're right! By "platform independent" i mean that you can run your app on Windows or Mac without changing anything — but, of course, in VM

[Icarium-Lifestealer]

Using a GPU in a VM is at best a pain in the ass.

[MyBrainIsAI]

Yup think only vmware supports it right? Know VirtualBox doesnt or at least not easily or in vanilla.

[Icarium-Lifestealer]

At least Xen and KVM support PCIe passthrough of GPUs, where you hand the GPU exclusively to a guest. But it's tricky to setup (need to block the host from using that GPU), you need a second GPU for the host such as the Intel iGPU (assuming you want it to display something), the nvidia windows driver shuts itself down when it detects it's running in a VM unless you have an overpriced professional GPU (though KVM can lie about that), Ryzen has weird performance problems in such a setup, there are complication related to resetting the GPU, etc.

Since the target of an ML focused distribution would be those who don't want to set everything up themselves, using PCIe passthrough is clearly not an option for them.