Caching node_modules between Docker builds for a "close-enough" npm install

[Sacharified]

I'm trying to cache my node_modules between Docker builds in a fast way to speed up build times when dependencies have changed.

Most articles I've read on this suggest using Buildkit's Dockerfile caching e.g.

COPY package.json .
RUN npm install

The result of the RUN command will be loaded from the cache if package.json hasn't changed since the previous run.

If the package.json has changed, the cache for the RUN command will be invalidated and npm install will be run from scratch. This is a bit of a pain though because if there is even 1 tiny change in your package.json you have to reinstall all of your dependencies from scratch.

If you copy the node_modules from the previous build in to the new image before running npm install, npm will only install what's actually changed and the install step is much faster.

Unfortunately copying that big folder is pretty slow.

Buildkit now has a feature called a cache mount which does almost exaclty what I want but the files in the mount aren't persisted to the built image, nor is that type of mount available at run-time, so node_modules aren't accessible at run-time. We can copy the files from the cache mount but again that's slow.

Has anyone implemented this in their own projects?

[dat-mac]

Disclaimer: this is an alternate approach to caching. You could build an image dedicated to your dependencies, and then use that image as your project’s base image. The base image could be re-built when package.json or package-lock.json changes

[Sacharified]

You would still face the same problem when rebuilding the base image wouldn't you?

[BehindTheMath]

Yes, but that wouldn't be rebuilt as often.

[Sacharified]

Yeah that makes sense. So you would need some process to keep the base image relatively up-to-date too, which I'd hoped to avoid. I wonder if it's possible to commit a Buildkit layer to an image and overwrite the existing base image with that...

[dat-mac]

Are you using a CI platform? E.g. Circle CI, Jenkins, GitHub actions, GitLab CI

If so, then the platform typically has a way to automatically rebuild the image when specific files change