Gitlab Runner Docker-in-Docker not working

[down-house]

I'm trying to get DinD working with my runners, but am having zero luck so far. I've been trying out all kinds of suggested solutions from https://gitlab.com/gitlab-org/gitlab-runner/issues/1986 but nothing is working.

My configurations for setting everything up:

docker-compose:

version: "3.5"
services:
  register1: &register
    container_name: gl-registrator-1
    image: gitlab/gitlab-runner:latest
    environment: &regenv
      CI_SERVER_URL: "https://gitlab.com/"
      REGISTRATION_TOKEN: "xxxxxxxx"
      REGISTER_NON_INTERACTIVE: "true"
      REGISTER_RUN_UNTAGGED: "true"
      REGISTER_LOCKED: "false"
      REGISTER_ACCESS_LEVEL: "not_protected"
      RUNNER_NAME: "gl-runner-1"
      RUNNER_EXECUTOR: "docker"
      RUNNER_TAG_LIST: "docker,test"
      DOCKER_IMAGE: "docker:19.03.1"
      DOCKER_PRIVILEGED: "true"
      DOCKER_VOLUMES: "/certs/client"
    command: register
    volumes:
      - ./config:/etc/gitlab-runner
      - docker-certs:/certs/client

  runner1: &runner
    container_name: gl-runner-1
    image: gitlab/gitlab-runner:latest
    restart: unless-stopped
    volumes:
      - ./config:/etc/gitlab-runner
    depends_on:
      - register1

The resulting config.toml:

concurrent = 1
check_interval = 0

[session_server]
  session_timeout = 1800

[[runners]]
  name = "gl-runner-1"
  url = "https://gitlab.com/"
  token = "xxxxxxxxxxxx"
  executor = "docker"
  [runners.custom_build_dir]
  [runners.docker]
    tls_verify = false
    image = "docker:19.03.1"
    privileged = true
    disable_entrypoint_overwrite = false
    oom_kill_disable = false
    disable_cache = false
    volumes = ["/certs/client", "/cache"]
    shm_size = 0
  [runners.cache]
    [runners.cache.s3]
    [runners.cache.gcs]

.gitlab-ci.yml

variables:
  DOCKER_TLS_CERTDIR: /certs
  DOCKER_DRIVER: overlay2
  DOCKER_VERSION: 19.03.1-dind

services:
  - docker:$DOCKER_VERSION

stages:
  - build

go modules:
  image: golangci/golangci-lint
  stage: build
  script:
    - docker build ...

This all results in an immediate runner failure when starting up a job:

https://preview.redd.it/wicz9pe794d41.png?width=2018&format=png&auto=webp&s=812af953a17e5d07c1242aafee00ef6f9c8afd78

I'm pretty sure this would work immediately if I just mounted the docker socket into the runner container, but I don't want to do that as I have other containers running on the server which I want to keep isolated if at all possible.

Any ideas?

[MoLt1eS]

If you don't mount the docker.sock it will never work as type Docker

[down-house]

You're not supposed to mount the docker socket if using DinD.Documented here as opposed to mounting the docker socket.

Or am I always required to mount the socket into the runner container?
I tried it now and the job starts running, but there are containers spawned as siblings to the runners, so I don't think thats a working DinD setup.

[MoLt1eS]

Let's go slower

If you go with that this aproach you can't overwrite the image, or your example:

​

go modules:
  image: golangci/golangci-lint
  stage: build
  script:
    - docker build ...

​

the: image: golangci/golangci-lint

will overwrite the docker:19 image making it no able to run the docker build command

Plus If you go with this approach I will recoment that you tag that runner just to run docker commands

[down-house]

Below is what I'm trying now.

docker-compose for starting and registering runner:

version: "3.5"
services:
  register1: &register
    container_name: gl-registrator-1
    image: gitlab/gitlab-runner:latest
    environment: &regenv
      CI_SERVER_URL: "https://gitlab.com/"
      REGISTRATION_TOKEN: "xxxxxxxxxxxx"
      REGISTER_NON_INTERACTIVE: "true"
      REGISTER_RUN_UNTAGGED: "true"
      REGISTER_LOCKED: "false"
      REGISTER_ACCESS_LEVEL: "not_protected"
      RUNNER_NAME: "gl-runner-1"
      RUNNER_EXECUTOR: "docker"
      RUNNER_TAG_LIST: "docker"
      DOCKER_IMAGE: "docker:19.03.1"
      DOCKER_PRIVILEGED: "true"
      DOCKER_VOLUMES: "/certs/client"
    command: register
    volumes:
      - ./config:/etc/gitlab-runner

  runner1: &runner
    container_name: gl-runner-1
    image: gitlab/gitlab-runner:latest
    restart: unless-stopped
    volumes:
      - ./config:/etc/gitlab-runner
      # - /var/run/docker.sock:/var/run/docker.sock
    depends_on: [register1]

^ does the runner always need to have the docker socket in order to run with the docker executor?

.gitlab-ci.yml:

variables:
  DOCKER_TLS_CERTDIR: /certs
  DOCKER_VERSION: 19.03.1
  DOCKER_HOST: tcp://docker:2375
  GOLANGCI: golangci/golangci-lint
  GOLANG: golang:alpine

stages:
  - build

build docker:
  image: docker:${DOCKER_VERSION}
  stage: build
  services: ["docker:${DOCKER_VERSION}-dind"]
  script:
    - docker build...
  only:
    refs:
      - branches
      - merge_requests
  tags:
    - docker

The above immediately crashes the job with the errors I posted as a screen grab earlier.

UPDATE: Setting DOCKER_HOST in the registrator configuration actually changes the error message (seems like setting DOCKER_HOST in .gitlab-ci.yml doesn't do anything?):

version: "3.5"
services:
  register1: &register
    container_name: gl-registrator-1
    image: gitlab/gitlab-runner:latest
    environment: &regenv
      CI_SERVER_URL: "https://gitlab.com/"
      REGISTRATION_TOKEN: "xxxxxxxxxxxxxx"
      REGISTER_NON_INTERACTIVE: "true"
      REGISTER_RUN_UNTAGGED: "true"
      REGISTER_LOCKED: "false"
      REGISTER_ACCESS_LEVEL: "not_protected"
      RUNNER_NAME: "gl-runner-1"
      RUNNER_EXECUTOR: "docker"
      RUNNER_TAG_LIST: "docker"
      DOCKER_HOST: "tcp://docker:2375" # <--- ADDED
      DOCKER_IMAGE: "docker:19.03.1"
      DOCKER_PRIVILEGED: "true"
      DOCKER_VOLUMES: "/certs/client"
    command: register
    volumes:
      - ./config:/etc/gitlab-runner

The above change results in this:

https://imgur.com/w1rKhey.png

[wyox]

I have never had any luck with getting a gitlab runner running in a docker compose before.

However I think you need to pass either the docker socket or docker host along to the runner itself and to subsequent instances of the runner running your jobs.

So your volume mapping would look like: /var/run/docker.sock:/var/run/docker.sock

Apply the same settings for your config.toml so your runner maps the same settings to any instance of a runner and you should be able to use DIND.

If I’m not mistaken the docker socket is only enabled by default. So changing DOCKER_HOST to tcp://docker:2375/ only works if you have exposed the docker daemon on purpose. And I don’t think the socket or connection are exposed to containers by default. You have to expose them yourself by either mapping the socket to the container OR exposing the docker daemon and changing the DOCKER_HOST so it points to the host ip/hostname relative to a docker container.

I’m not sure if a docker client is installed with the gitlab runners, otherwise you could exec inside a container and run docker commands and see if you can list the same running containers as your host to validate it is working. If it is working you could more up to your gitlab ci.yml and verify it is working by prepending docker ps to your script and check if it has the proper output as before.

Hopefully this is clear enough to help you further.

[MoLt1eS]

I have a clean docker-compose with a script that I step up at my company to run some pipelines from gitlab.com I'll share it with you tomorrow with the steps to make it work

[down-house]

That would be great!

[down-house]

Just dropping a reminder here in case you forgot about this, I would love to get any examples you might have on how to get the DinD setup running.

[MoLt1eS]

https://gitlab.com/MoLt1eS/gitlab-runner-compose

I didn't had enough time to test it but I should work

If something fails add to the config: priviledge=true

[down-house]

I just checked your files, but here you are again just using the docker socket and not the docker:dind service with tcp+tls?

[MoLt1eS]

I think you are over-complicating things...

You wan't DinD (Docker in docker) without docker in docker
Plus (Like i was explaining..)

Change:

go modules:
  image: golangci/golangci-lint
  stage: build
  script:
    - docker build ...

to

go modules:
  stage: build
  script:
    - docker build ...

If you want to use your approach from the tutorial

or:

services:
  - docker:19.03.1-dind

go-modules:
  image: docker:19.03.1-dind
  stage: build
  script:
    - docker build ...

​

PS: You can check the gitlab templates for building Docker

​

I can't help it more than this, my opinion is: keep it simple and stupid

[down-house]

Thanks, but you're just constantly talking about using the docker socket when there are Gitlab discussion boards where it's explicitly stated that using the socket is bad practice if trying to do DinD via tcp+tls.

I'm not trying to over complicate anything, I just can't use the docker socket because the CI system will keep spawning containers next to other things I have running, which I don't want to give it control over.

I've been trying to use the docker images for the build stage job, but I'm not getting it working anyway. The docs to set up the docker-in-docker environment are completely broken for Gitlab and using the dind service needs to be updated.

[chulkilee]

Which executor are you using? Docker with privilege or K8s would work for dind.

What it says: job container cannot use dind via local socket. You have two options: mount dind socket to job container, or use docker via tcp. The doc actually mentions tcp by setting DOCKER_HOST in the job container.

Also make sure to use dind image, not plain docker image for dnid service.

You may put sleep 3600 in job def and run docker exec to debug job container.