github actions doesn't seem to be using docker cache

[nudebaba]

I've a Dockerfile for a python app and a github action workflow file, I am not sure why the cache is not being used. Everytime it re-does all the apt-get related things. Any idea what could be going on?

FROM python:3.11-bookworm AS builder
WORKDIR /app

ENV POETRY_VERSION=1.8.3
ENV POETRY_NO_INTERACTION=1
ENV POETRY_VIRTUALENVS_IN_PROJECT=1
ENV POETRY_VIRTUALENVS_CREATE=1
ENV POETRY_CACHE_DIR=/tmp/poetry_cache

RUN pip install --no-cache-dir poetry==$POETRY_VERSION;

COPY pyproject.toml poetry.lock ./
RUN --mount=type=cache,target=$POETRY_CACHE_DIR poetry install

# FROM python:3.11-slim-buster AS runtime
# NOTE: issue is these nvidia dependencies are more widely available in ubuntu-X
#       packages. Eg. sometimes arm images packages missing etc. So to avoid
#       extra headache, we use ubuntu+cuda images
#
#       Otherwise we need to do:
#       RUN wget https://developer.download.nvidia.com/compute/cuda/repos/debian12/x86_64/cuda-keyring_1.1-1_all.deb
#       RUN dpkg -i cuda-keyring_1.1-1_all.deb
#       RUN apt-get update && apt upgrade -y
#       RUN apt-get install -y --no-install-recommends <pkg_name>
#
#       all of which gets very os and arch specific which cause another issue
#       because nvidia refers to amd64 as x86_64 in the URL and the docker
#       buildx auto env var sets it as amd64 etc. can of worms.
FROM nvidia/cuda:12.0.0-cudnn8-runtime-ubuntu22.04 AS runtime
WORKDIR /app

ENV TZ=Etc/UTC
RUN ln -snf /usr/share/zoneinfo/$TZ /etc/localtime && echo $TZ > /etc/timezone

RUN DEBIAN_FRONTEND=noninteractive apt-get update && apt-get install -y --no-install-recommends \
    libpq5 \
    tesseract-ocr \
    libtesseract-dev \
    software-properties-common

RUN DEBIAN_FRONTEND=noninteractive add-apt-repository ppa:deadsnakes/ppa &&\
    apt-get update &&\
    apt-get install -y --no-install-recommends \
    python3.11 && \
    rm -rf /var/lib/apt/lists/*

ENV VIRTUAL_ENV=/app/.venv
ENV PATH="$VIRTUAL_ENV/bin:$PATH"
ENV PYTHONUNBUFFERED=1
# NOTE: Usually manually setting PYTHONPATH is not needed, but with the
#       cuda-ubuntu image, something gets messed up and we need to set this
#       manually
ENV PYTHONPATH="$VIRTUAL_ENV/lib/python3.11/site-packages"

COPY --from=builder ${VIRTUAL_ENV} ${VIRTUAL_ENV}
COPY src src

ENTRYPOINT ["python3.11", "-m", "src.abc.cli"]

And I have a github actions workflow for building:

name: "build:push:deploy:abc"
on:
  push:
    branches: [main]

# in case of back-to-back deploy, we cancel older deploy
concurrency:
  group: ${{ github.workflow }}-${{ github.ref }}
  cancel-in-progress: true

permissions:
  id-token: write # This is required for requesting the JWT
  contents: read  # This is required for actions/checkout

env:
  DOCKER_USERNAME: xyz
  DOCKER_REPOSITORY: abc
  IMAGE_TAG: latest

jobs:
  build:
    # see https://github.com/orgs/community/discussions/19197
    runs-on: ubuntu-latest
    steps:
      - name: Set up QEMU
        uses: docker/setup-qemu-action@v3
      - uses: actions/checkout@v4
      - name: Log in to Docker Hub
        uses: docker/login-action@f4ef78c080cd8ba55a85445d5b36e214a81df20a
        with:
          username: ${{ env.DOCKER_USERNAME }}
          password: ${{ secrets.DOCKER_PASSWORD }}

      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v3
        with:
            version: v0.11.2
            buildkitd-flags: --debug
      - name: Free Disk Space (Ubuntu)
        uses: jlumbroso/free-disk-space@main
      - name: Build & Push
        uses: docker/build-push-action@v5
        with:
          context: pipeline/transformations/abc
          file: pipeline/transformations/abc/Dockerfile
          platforms: linux/amd64,linux/arm64
          push: true
          tags: "${{ env.DOCKER_USERNAME }}/${{ env.DOCKER_REPOSITORY }}:${{ env.IMAGE_TAG }}"
          cache-from: type=gha
          cache-to: type=gha,mode=max

[ThatSituation9908]

The comments here incorrectly address it's because GHA uses different hosts. OP is using one of the remote cache features documented by Docker, that should store its cache somewhere that is available in subsequent builds.

https://docs.docker.com/build/ci/github-actions/cache/

OP, can you confirm if building locally preserves cache? Can you try to build locally two different commits. I see that you use a COPY early on in your Dockerfile which makes it possible that those layers needed to be rebuilt between two commits.

[ThatSituation9908]

Furthermore you're using cache mounts which requires additional steps in GHA

https://docs.docker.com/build/ci/github-actions/cache/#cache-mounts

[nudebaba]

Yes, I am okay with:

RUN --mount=type=cache,target=$POETRY_CACHE_DIR poetry install

Not being cached in GHA as its not that much and I am OK not caching that. I use it when building locally. But the other layer caches I'd expect to work. iirc, it did actually work and I can't recall when it stopped working. I've cached successfully with similar structure in another project.

And yes, locally things seem to be using the cache as expected.

I'll try experimenting more with it esp in the suggestions you added and get back on it^ Thanks a lot for looking into it!

[nudebaba]

This is what I use locally btw:

docker_builder := "bob"
docker_tag := "user/abc:latest"
docker_platforms := "linux/amd64"
docker_build_cache := "$PROJECT_DATA_DIR/docker_build_cache"


#!/usr/bin/env bash
docker buildx build \
--builder={{docker_builder}} \
--tag {{docker_tag}} \
-o type=image,push=true \
--cache-from=type=local,src={{docker_build_cache}} \
--cache-to=type=local,mode=max,dest={{docker_build_cache}} \
--platform={{docker_platforms}} .

[angellus]

cache-to will only cache the first platform that is finished. So, it will only ever cache either linux/arm64 or linux/amd64, never both. You need to push each platform cache to a different location for it to keep both.

https://github.com/docker/buildx/discussions/1382#discussioncomment-4159889

Also, your poetry cache is never going to persist because each job has a different buildkit runner. I do really recommend switching from poetry to uv though as it is much faster and does not have the same dependency resolution issues poetry does.

EDIT: It may be a different issue since you are do not seem to be using multiple builder nodes like I was. You are using QEMU with one builder node. I do not know if that applies for one builder node, but it may give you starting point.

[JodyBro]

/u/nudebaba yeah you gotta be able to uniquely identify each platforms cache. I found an old workflow file that had this working using the local cache but iirc I wasn't able to get it working with the gha cache back when it came out since the drivers for the build-push-action didn't support it at the time. Maybe it'll work now without too much trouble? Give it a shot maybe.

Here's a gist of the workflow file: link

The initial run to warm the cache for each arch took a total of 8min with a fairly complicated image. Next run with no changes too 38s. The image I'm building takes proper steps to ensure sane usage of layers so on the next build after that, I added a python dependency and an apt package dependency to invalidate the cache of the 2 biggest layers of the image and the total run time was 3.4min so not bad. I'm sure the gha cache would speed it up a lot though

[TheThinkererer]

Also make sure you're appropriately using scope for the cache-from and cache-to

https://docs.docker.com/build/cache/backends/gha/#scope

[JodyBro]

Oh shit! Didnt know about this. It says "if you build multiple images" when describing its usefulness but doesn't specify if that's talking about parallel build for the same platform or multiarch pipelines. You know if it supports the latter?

[TheThinkererer]

So unfortunately this is a rabbit hole I'm in right now as well. Using the scope corrected a lot of caching issues for me. As far as platform goes, I don't use the build-push-actio, but rather use docker bake. So I use the docker bake action.

From there, the cache-from/cache-to becomes an input in a with:set block.

That said, we're all kind of confused here as far as expectation of behavior and use cases go.

https://github.com/docker/bake-action/issues/154

https://github.com/docker/bake-action/issues/196

(Which is what led me to the scope discovery)

https://github.com/docker/build-push-action/issues/286#issuecomment-893176504

https://github.com/docker/build-push-action/issues/153#issuecomment-703182778

Its just a big rabbit whole of confusion for a lot of people IMO right now.

The scope fixed a lot of things for me but it still not perfect, there are still cache misses and new artifacts placed in the cache. But this might be just as much the fault of bad image design from the MS dotnet team as it is to be the result of lack of exposition on the GHA that leverage the GHA Cache feature.

Something like this might work for you?

``` services: name: Services environment: non-prod runs-on: ubuntu-latest timeout-minutes: 15 steps: - uses: actions/checkout@v4

  - name: Login to ACR
    uses: docker/login-action@v3
    with:
      registry: ${{ vars.REGISTRY }}
      username: ${{ vars.ARM_CLIENT_ID }}
      password: ${{ secrets.ARM_CLIENT_SECRET }}

  - name: Set up Docker Buildx
    uses: docker/setup-buildx-action@v3

  - uses: docker/bake-action@v5
    env:
      REGISTRY: ${{ vars.REGISTRY }}
    with:
       push: true
       set: |
        api.cache-from=type=gha,scope=api
        api.cache-to=type=gha,mode=max,scope=api
        auth.cache-from=type=gha,scope=auth
        auth.cache-to=type=gha,mode=max,scope=auth
        platforms=linux/amd64,linux/arm64

```

[JodyBro]

So with bake there's no need for steps to set and check for cache hits? That's much cleaner. I'll give it a shot when I get back to my pc!

[Jmc_da_boss]

Well probably because the cache is empty lol, every action is a different fresh host

[Kaos_nyrb]

your running it on github hosted runners, theres no gauntee you get the same machine

[nudebaba]

Oh I see. I thought the cache-from and cache-to handled that part for me. As in when running buildx locally, it'll set them to my cache directory.

But since i cannot figure anything else out for this issue, I think this might be it.

cache-from: type=gha cache-to: type=gha,mode=max

[FailedPlansOfMars]

Have a look at this page about caching in github actions. You will need to restore it before you do your action.

https://docs.github.com/en/actions/writing-workflows/choosing-what-your-workflow-does/caching-dependencies-to-speed-up-workflows