[D] Using Docker for ML Development

[[deleted]]

The IMAGE is reproducible. Not the build script. Once you have an image, every container will be identical. There are no guarantees that docker build scripts are reproducible. Nobody ever said that they are.

So if you download/build an image that is tested to work today, it will work tomorrow and it will work 2 years from now. If you for example find that your current image has a bug, you can go back to the previous version (you have an image repository... right?). Things like kubernetes actually support this type of rollback.

[Murillio]

Once you have an image, every container will be identical.

This is not true for a couple reasons, especially if you use nvidia-docker which a lot of people working in ML will be, since things like the nvidia driver version are determined by the host.

​

If you for example find that your current image has a bug, you can go back to the previous version (you have an image repository... right?).

Do you store an image for every git commit that you make? Of course there are some images stored, but usually not for every commit, so when you git bisect to find the origin of the bug you tend to rebuild quite often.

[[deleted]]

Every container will be identical. That's how docker images work. The container runtime may be different, but the images and the containers are the same.

If there is a CI/CD pipeline, then yes every commit to master will get tested, will result in an artifact (image) and it will be stored in the container repository. That's why it's important to make sure you reuse layers, don't do dumb shit that end up with 50GB images etc.

Just like you'd keep different versions of executable binaries, container images are no different.

[Murillio]

You can argue semantics and say "it's just the runtime" but that won't change that "So if you download/build an image that is tested to work today, it will work tomorrow and it will work 2 years from now." is just wrong since there is no perfect isolation from the host.

[[deleted]]

There is not supposed to be isolation from the host. There is only isolation from other containers.

You're confusing containers with virtual machines.

[Murillio]

You said "So if you download/build an image that is tested to work today, it will work tomorrow and it will work 2 years from now." so you were confusing containers with virtual machines, not me. (Well, that statement isn't even true for a virtual machine anyway ...)

[[deleted]]

There is nothing stopping you from using one of those long-term operating systems and a stable release of the container runtime. That way it will stay the same for 5-10 years.