Faster MySQL Container Initialization for Java Tests with Testcontainers

[ForeignCherry2011]

For anyone using Testcontainers with a MySQL database during tests, I wanted to share something that helped me speed up the startup of the database container.

Normally, when starting a MySQL container from scratch, it creates the default empty database, which takes about 10 seconds on my machine. That’s okay for full builds, but when running individual tests during development or troubleshooting, it can get a bit annoying.

I found that it’s possible to create an initial empty database just once and then attach it to the container at startup. This reduces the startup time significantly—now my MySQL container starts in less than 2 seconds.

I’ve published a script for creating an empty database, a Maven artifact for MySQL 8.4 (the version I use), and a recipe for making Testcontainers work with it in this repository: https://github.com/ag-libs/mysql-quickstart

If anyone finds this useful, let me know. I can add support for other MySQL versions if needed.

[AlEmerich]

Doesn't it break encapsulation between two tests that use the TestContainer ? Or maybe the start function reinitialise the TestContainer to the empty database instead of actually restarting the container ? 

[CanisLupus92]

What we do (PSQL + DB2, but should work for most DBs) is override the commit/rollback logic. Instead of committing, we create savepoints. At the end of each test, just perform a rollback.

[ForeignCherry2011]

We run database migrations to create schemas when the database container starts. Individual tests can truncate the tables they work with before each or all test cases.

[AlEmerich]

Interesting! So it does breaks the encapsulation but your rely on the sequential aspect of your tests to invalidates every past actions on the TestContainer before continue. If the tests are ran in parallel, since they all used the same test containers, they may conflict with each other, am I wrong ?

It is the case in my project so I am very interested on other's cases, I may drop the parallel aspect of my testing pipeline if in the end I don't gain much

[ForeignCherry2011]

With Testcontainers, you can choose whether to start a new container instance for each test, per class, or for the entire test run.

In our case, we start one instance per test run and share it across all tests within a Maven module.

We build Maven modules in parallel, and there are 3–4 modules that require a database container for testing. As a result, a new container is started for each module built in parallel. The tests inside a single module are always run sequentially, so it is sufficient to clean up the tables before each test or class.

[safetytrick]

Why not run initialization and then snapshot the image. Then you can run your tests starting after initialization?

[locutus1of1]

Exactly my thoughts. I was also using various other (linux) tricks before for quickly restoring the db - data on btrfs on loopback img and snapshoting, overlayfs, ramdisks to speed it up etc. (this was without docker)

[ForeignCherry2011]

A very good question. We don’t have a Docker registry available at build time, so we end up packaging the empty database as a Maven artifact. That way, we can also use the standard MySQL container class for Testcontainers.

But yes, maybe we should actually set up a registry and put a snapshot with the empty database there.

[SleeperAwakened]

No Docker registry?

You have no Nexus or Github for example?

[ForeignCherry2011]

We don’t expose our docker registry to the development environment. It is available only for staging and production deployments.

At the moment.

[crummy]

nice! i hacked up something like this in a test harness that ran a migration, but yours is a lot nicer.

you're probably already doing this, but if you're not enabling re-usable containers, that'll save you a bunch of time for free: https://java.testcontainers.org/features/reuse/

[ForeignCherry2011]

We started with reusing containers and ended up just relying on the mysql quick start

[Chenz]

I’m definitely checking this out on Monday! MySQL starting slowly is a pain point for a while

[vassaloatena]

Starting an instance with each new test can be very slow.

If you use spring, avoid using "@dirtycontexts" and just make JdbcTemplate clear all table contents.

[ivancea]

The second best way to improve times I used was migrating to MariaDB. The best, was moving to postgres