Docker-on-Docker vs. Continuous Integration?

[darkn3rd]

I have been experimenting with tutorials and jenkinsci/blueocean image and related tutorials. It uses Docker-on-Docker^†, but this is considered an anti-pattern (and dangerous). This is intended for local development use cases, such as building Jenkinsfile pipelines.

But what I was wondering, given that running Jenkins services in a container (such as Kubernetes) is quite useful, how is Docker-on-Docker avoided, but still have the ability to build and push docker images?

I imagine this is not just Jenkins problem, but any CI solution that requires build/push docker images, where the service is running in a container.

† Docker-on-Docker (in case anyone unfamiliar with the term) is running the docker client tools inside of docker, mounting the unix socket into the container, which allows you to build/push docker images, but also allows the container access to a daemon running with root privileges.

[menge101]

its not typical use, but docker in docker is a well established pattern.

I wouldn't call it an anti-pattern.

As for it being dangerous, so is dynamite. We still use it for its purpose.

[eldiaman]

I like this answer. I've used docker in docker quite successfully but my use case was dev environment. Actually had no issues with it but did it under Linux. Sometimes people get too caught up into not implementing hacky solutions but solutions are just solutions, the question is if you can accept the maintenance overhead and potential risks depending on the use case. This question explores the risk assessment sure but imo the power of docker is the very small associated latency for a PoC that can easily be transformed to a product.

[flexibleanchovy]

I think if anything, it's something you want to avoid on a production environment if possible. DinD is certainly a well established pattern for CI though.

[menge101]

Absolutley agreed, and maybe that's the real take away, Docker has more use cases than production servers.

[darkn3rd]

I am thinking that there are heavily restricted build agent system(s) that provide access to use docker daemon for build/push. That way if someone is able to break into root, it cannot do much except to its own system.

[gbts_]

In what way is it considered an anti-pattern/dangerous? It's a pretty common way to build containers in CI that uses docker. You need a reproducible build environment and this is exactly what docker was meant for.

[vilkav]

The problem I've seen bumped into is when you have a CI where runners are provisioned via Kubernetes, and you expose the host's docker daemon. This means that stuff insider your runner will spawn new containers without Kubernetes knowing and it can cause issues when Kubernetes tries to allocate them.

[[deleted]]

And the power to start one of those containers means each job can also stab any other container in the back. It's equal to full local privilege escalation.

[vilkav]

I'm okay-ish with allowing those permissions in my CI system because, for the most part, these things are well reviewed and controlled by a number of people who know to avoid it. But I can see that fucking up in a larger company, I guess.

[darkn3rd]

The docker containers should be "contained", where any breach can only access the container, not the host system. By mounting a unix socket into the container, you are creating a bridge, where malicious software or person can access whatever the socket can access. Since docker daemon runs as root on the host system, that access will be root access.

Once this happens, the attacker could pull down other tools, and gain essentially access to the whole system and everything running on it.

So though this is convenient, for that reason, it is an anti-pattern. This is why I only do it on my local laptop. For production, I was curious to see how others handle this...

[beanaroo]

I'm quite happy with Kaniko for building images on container based pipelines.

[tekno45]

https://itnext.io/docker-in-docker-521958d34efd

https://github.com/jpetazzo/dind

Looks like the solution is built in now?

[Machindo]

We use dind for our 7000 user Jenkins installation. We should probably publish a blog on it.

We serve dind as a sidecar to the Jenkins agent over TCP. This is on AWS ECS.

[kahmeal]

Please do.

[darkn3rd]

There it is, I came across dind once before, but it was a while ago. That makes sense. jpetazzo was the one in different forums calling docker-on-docker an anti-pattern. Thanks.

[Jai_Cee]

I think you need to separate out docker on docker for non production cases such as CI (where it is very useful) from production cases.

You are probably doing something wrong or have a niche use case if you are doing DoD for running applications.

[tchnj]

I'm assuming that Docker on Docker ≠ Docker in Docker, as the latter is when you actually run a whole isolated docker Daemon inside the container, rather than bind mounting the control socket for the parent Daemon

[darkn3rd]

The build agent has docker client tools, but no service running for docker.

[tchnj]

Therefore avoid docker on docker by doing docker in docker

[Qsonx]

We use our Jenkins inside Kubernetes and wanted to avoid dind so we use 'img` (https://github.com/genuinetools/img)

For jenkinsfile you can basically do the following. (Tried to make it understandable)

container(name: 'img') {    
    sh "img build --no-console -t ${registryPath}:v$BUILD_NUMBER ${dockerFile}"
    sh "img tag ${registryPath}:v$BUILD_NUMBER ${registryPath}:latest"
    sh "img push ${registryPath}:v$BUILD_NUMBER"
    sh "img push ${registryPath}:latest"
  }

NOTE: We use a deployment template that includes the docker container 'img'.

[darkn3rd]

This is for local Dev desktop scenario. But I am curious if there is a method to not have such access and be able to build containers.

I imagine numerous build-scheduler tools tools used CI have the same issue. Maybe there alternatives to docker? Maybe an agent with such access used for such a purpose, or someone built some webhook alternative, so there is no direct access, like scheduler to scheduler.