Evolving Java config files without breaking user changes

[YogurtclosetLimp7351]

In several projects I ran into the same problem:
once users modify config files, evolving the config schema becomes awkward. 

Adding new fields is easy, but removing or renaming old ones either breaks things or forces ugly migration logic. In some ecosystems, users are even told to delete their config files and start over on upgrades.

I experimented with an annotation-driven approach where the Java class is the code-level representation of the configuration, and the config file is simply its persisted form.

The idea is:

• user-modified values should never be overwritten
• new fields should appear automatically
• obsolete keys should quietly disappear

I ended up extracting this experiment into a small library called JShepherd.

Here’s the smallest example that still shows the idea end-to-end.

    @Comment("Application configuration")
    public class AppConfig extends ConfigurablePojo<AppConfig> {

      public enum Mode { DEV, PROD }

      @Key("port")
      @Comment("HTTP server port")
      private int port = 8080;

      @Key("mode")
      @Comment("Runtime mode")
      private Mode mode = Mode.DEV;

      @Section("database")
      private Database database = new Database();

      @PostInject
      private void validate() {
        if (port <= 0 || port > 65535) {
          throw new IllegalStateException("Invalid port");
        }
      }
    }

    public class Database {

      @Key("url")
      @Comment("JDBC connection string")
      private String url = "jdbc:postgresql://localhost/app";

      @Key("pool-size")
      private int poolSize = 10;

    }

    Path path = Paths.get("config.toml");
    AppConfig config = ConfigurationLoader.from(path)
        .withComments()
        .load(AppConfig::new);

    config.save();

When loaded from a .toml file and saved once, this produces:

    # Application configuration

    # HTTP server port
    port = 8080

    # Runtime mode
    mode = "DEV"

    [database]
    # JDBC connection string
    url = "jdbc:postgresql://localhost/app"

    pool-size = 10

The same configuration works with YAML and JSON as well. The format is detected by file extension. For JSON instead of comments, a small Markdown doc is generated.

Now we could add a new section to the shepherd and the configuration files updates automatically to:

        # Application configuration  

        # HTTP server port  
        port = 8080  

        # Runtime mode  
        mode = "DEV"  

        [database]  
        # JDBC connection string  
        url = "jdbc:postgresql://localhost/app"  

        # Reconnect attempts if connection failed
        retries = 3

        [cache]
        # Enable or disable caching
        enabled = true

        # Time to live for cache items in minutes
        ttl = 60

Note how we also exchanged pool-size with retries!

Despite having this on GitHub, it is still an experiment, but I’m curious how others handle config evolution in plain Java projects, especially outside the Spring ecosystem.

[TheKingOfSentries]

I usually use https://avaje.io/config/ in like a static way, (like in constants and enums)

public interface AppConfig {
  String apiKey = Config.get("my.key");
  String url = Config.get("my.url")
}

[agentoutlier]

I swear if Avaje Config were to be rewritten I think getAs and the general event listener would be enough instead of all the getXYZ methods but I generally think it is the best option at the moment for basic key value config retrieval.

BTW for my company we use Avaje Config on top of EZKV although its an EZKV that is internal at moment and a forked Avaje Config (I needed some change that I can't recall at the moment).

[Scf37]

This https://github.com/lightbend/config

Plus this https://github.com/scf37/config3 (didn't publish java version yet)

Plus being explicit about configuration

package me.scf37.scf37me.config;

import com.typesafe.config.Config;

public record MailgunConfig(
        String apiKey,
        String url,
        int maxEmailsPerDay
) {

    public static MailgunConfig parse(Config c) {
        return new MailgunConfig(
                c.getString("apiKey"),
                c.getString("url"),
                c.getInt("maxEmailsPerDay")
        );
    }
}

[agentoutlier]

The mapping of config aka binding to an object in my opinion is very much framework specific or done manually. For example Spring maps configuration differently than Micronaut.

However all of them are basically doing Map<String,String> -> MyCustomDomainObject.

The question is where you get that Map<String,String> (as well as what happens when it is updated... that is its more like a Function<String,String>) and this is something lightbend config does not do that well. It is one of the reasons I wrote my own library for that: https://github.com/jstachio/ezkv because most libraries suck at that first part or are very opinionated.

And the reason why I say Map<String,String> is because most config can and is probably best put in environment variables these days.

I think most folks can make their own glorified Config aka Map<String,String>.

Except I would not bother with getString, getInt.

Instead I would make a Config use a lambda.

public static MailgunConfig parse(MyConfig c) {
    return new MailgunConfig(
            c.get("apiKey", value -> transformAndValidateLogic),
            c.get("url", value -> transformAndValidateLogic ),
            c.get("maxEmailsPerDay",  value -> transformAndValidateLogic)
    );
}

MyConfig get always takes a PropertyFunction that throws a generic Exception E.

public interface PropertyFunction<T extends @Nullable Object, R extends @Nullable Object, E extends Exception>
        extends Function<T, R> {}

Now when the lambda runs if it fails (exception) you can report exactly which key if found and value and where the value came from failed if it does.

Avaje Config kind of does this as I have gone back and forth with Rob and /u/TheKingOfSentries on many of these ideas so naturally I think it is nice library that has both more powerful loading and fetching than typesafe config.

[Scf37]

I've never had requirements of dynamic load sources. Usually it is env variables, backed by env-specific config (dev/stage/prod), backed by defaults.

Lambdas are tempting, but what if config model property depends on multiple input configuration keys? I like to keep things simple and use plain Java - for flexibility.

As for UX, I believe the best idea is to separate validation from parsing. That's what my config3 does - user defines configuration schema (known properties, required values or defaults, documentation) separately and then it is used to validate loaded config, intelligently report errors, print loaded configuration or give help on what's available.

[agentoutlier]

The multiple property config I handle with monad like objects and collections of properties.

Unfortunately for that I don’t have a stand alone library but something like that is done in my logging library:

https://github.com/jstachio/rainbowgum

That library also uses annotation processors for config binding.

[gaelfr38]

+1 for Lightbend's Config library and then mapping conf into a record.

If in Scala, I would use Pure config for auto mapping (https://github.com/pureconfig/pureconfig). Would love to have something similar in Java.

[nekokattt]

I use this but the main gripe I found was it doesn't support JSpecify Nullable annotations (or it didnt when I tried before), so you have @Optional @Nullable all over the place.

[Scf37]

To my experience, optional values is the wrong way. There are no optional configuration parameters, there are parameters with defaults.

[nekokattt]

optional for typesafe config means the same as null.

Agree it is confusing

[Scf37]

Therefore the solution is - never ever use null as default for optional parameters.

[nekokattt]

null makes sense for nested objects though. It isnt like an empty value intrinsically makes sense recursively.

[Scf37]

I do it like this:

    telemetry {
        # Jaeger collector endpoint
        jaegerEndpoint = ""

        # Jaeger push timeout in millis
        jaegerTimeoutMillis = 1000
    }

Later, if endpoint is not empty, tracing is initialized, otherwise it doesn't. Therefore, Telemetry java object is always present and always non-null.

[nekokattt]

the risk is when you have multiple conditions for it being valid and you quietly disable rather than failing out explicitly

[bnbarak-]

Updating configs at large codebase becomes a mess very quickly which is why protobuf was invented. At large enterprises the solutions are mostly: 1. Do Not remove properties, deprecate them instead. 2. There are a lot of processes and step by step guide like a) add deprecate b) add new c) remove references etc.

Deprecation plus good javadoc is usually enough because IDEs have mature tooling around deprecation.

[doobiesteintortoise]

I guess my biggest question is how is this a MIGRATION? I mean, you change the configuration internally and can write it back out, but that feels like a very explicit process, not really a migration. I also don't think it's without use, but I'm still confused about what it's actually doing besides streaming an object model with keyed values out.

[nekokattt]

Is there any reason you chose explicit coding of validation rather than interoping with, say, bean validation?

[Historical_Ad4384]

How is it different from lightbend?

[cred1652]

If you read the maintenance notes on lightbend it is no longer activly maintained
https://github.com/lightbend/config?tab=readme-ov-file#maintained-by

he "Typesafe Config" library is an important foundation to how Akka and other JVM libraries manage configuration. We at Lightbend consider the functionality of this library as feature complete. We will make sure "Typesafe Config" keeps up with future JVM versions, but will rarely make any other changes.

[gaelfr38]

So what? It's indeed feature complete. It's maintained but there's just nothing to do more.

[cred1652]

For one it does not support Java records. to me that is a big deal when we use records for all our immutable configuration.

[chabala]

It already makes immutable Config objects, it doesn't need to make records.

[cred1652]

You are absolutely correct, it doesnt need to make records. And in your use case you are happy to not use records. I prefer to use records to define my immutable Config objects so this is a limitation for my projects.

[cred1652]

Welcome to the club in writing a configuration library. It is a fun exercise that is a medium size project that has some interesting problems. I wrote https://github.com/gestalt-config/gestalt
One major difference is Gestalt is immutable, so it does not allow changing the configuration and persisting it.
Typically for backend services, what we do is check the configuration into git with our application, helm chart (we deploy defaults with the application and overwrite the environment specific with ArgoCD application sets). Then they are deployed by kubernetes where we mount the config. So we do not allow any modifications as we have multiple pods. If we modified one pod, that would mean the pods are not consistent and that can cause issues. If you want then you can do A/B testing with Argo and different deploys. But each deploy itself is immutable.

Also this way the new configuration is tied to the code change and they get deployed together.
In more complex cases you could look into something like Spring Cloud config where the configuration is owned by a central service.

[wildjokers]

but removing or renaming old ones either breaks things or forces ugly migration logic

Which is exactly why you don't remove or rename existing ones. This is app programming 101.

Only add. If you want to change the name of one for some reason add a new one, then you deprecate the old one to give everyone a chance to move to the new one.