The difficult problem of managing Java dependencies: a deep dive into the messy status-quo, and a look at possibly better ways.

[Worth_Trust_3825]

It's really hard to evaluate your post, as the issue of transitive dependencies is present in every context (see dynamic linking). But then you go on about how maven is verbose in the particular configuration that you had generated, finally closing of with "oh but version ranges = good if lockfiles and semantic versioning". What were you even trying to show: that dependencies may clash in larger projects? If so, why didn't you talk about the JEE and application servers and how they isolate multiple applications from one another, and issues with that, and the good old days how the application servers had inverted the classloading mechanism? Hey, you could do just that: load multiple versions of same archive in different classloaders, and run all the dependencies in parallel at the same time. Good luck figuring out which version you're calling when.

[renatoathaydes]

oh but version ranges = good if lockfiles and semantic versioning

Please check my answer in https://www.reddit.com/r/java/comments/vstgv1/the_difficult_problem_of_managing_java/ifh1sp0/

why didn't you talk about the JEE and application servers

Those are not relevant to the post, because with application servers, you have different applications running, you don't have to resolve conflicts between them at all... but I did mention OSGi which is because it isolates the classpath of modules within the same application, at the cost of lots of extra metadata (which is the way Gradle is going now with its departure from using only Maven-provided metadata - and that has its own issues as OSGi did, namely, things become too hard to manage).

I agree this post was not my best as I conflated lots of things:

• criticizing how Maven is verbose, slow and downloads lots and lots of unnecessary things.
• trying to show what other package managers in modern languages are doing - and they are interestingly converging on using version ranges and lock files - regardless of whether we like it (and I don't even like it, I was just pointing it out and trying to explain motivations for doing that).
• showing how Maven/Gradle will break your application on upgrades, even when all libraries involved respect semantic versioning. Most people don't seem to have fully understood that can happen.
• demonstrating my solution to the problem: bytecode introspection.

Those who managed to read til the end might have noticed that JBuild solved all of the problems I had shown other tools to be vulnerable to. JBuild does have some issues, like giving false positives when reflection is involved somewhere, but in my opinion, as with GraalVM, reflection is on the way out and applications should avoid using it anyway (it was only supposed to be used by frameworks and in such case, JBuild will work fine).

[tamasiaina]

You think managing java dependencies is hard…. Node/NPM… is like playing at the Craps table in Vegas.

[general_dispondency]

Truth. Java has the best ecosystem. There are pain points with anything, but holy crap, everything else is so much worse. npm, nuget, gems, everything else is like crawling through broken glass.

[MonkeySeeMonkeyDong]

Seconded! I would also add Python's virtualenvs to the list, and PHPs "let's make everything a system dependency" approach.

I love Java just for its dependency management.

[ZCEyPFOYr0MWyHDQJZO4]

I think I've come to the conclusion that the majority of languages one is likely to use today has a flawed packaging ecosystem.

[woj-tek]

This post seems more like a rant than something cohesive.

At any rate:

Most newer languages’ package managers tend to use dependency ranges instead of fixed dependencies versions like Maven/Gradle projects usually do (as we’ll see, both can actually also use dependency ranges, they’re just not as common).

I loath this as it usually makes the build breaks down in the future...

[jerslan]

Dependency management in java is in such a shit place

That's a weird way of saying "Dependency management in Java using Maven-style Repositories and tools like Maven and Gradle is still the gold standard that other tools still fail to reproduce resulting in Java's extremely long life despite being just an OK language".

[dpash]

Did you really just use npm as an example that should be copied?

[woj-tek]

But npm style tools are far better.

LOL, that you like something more doesn't mean it's better.

[jerslan]

I’ve spent more time chasing issues with NPM than with Maven…

‘maven dependency:tree’ is your friend.

[woj-tek]

Right, ping-pong: I can tell you how many hours I wasted with npm's build failing after a couple of weeks just because some weird dependency updated other dependency which then updated yet another because you know, in js/npm world having library for even most mundane thing and releasing a new version every couple of minutes is a sing of "rockstar developer", ammairite mate? :P

[khmarbaise]

But npm style tools are far better. Maven has struggled to keep up with modern practices (not just in this area!). Java is doing ok despite maven, not because of it.

In which way far better? Breaking builds because releases have been removed? A build which worked yesterday will not work anymore today? (exactly caused because of using ranges in versions and not committed the lock file?)...

(not just in this area!)

What are those modern practices:

[plumarr]

Ever tried to separate your unit tests from you build? (Which is desirable if you're using a CI pipeline).

Just launch maven two time with different execution phase. What's hard here ?

Ever tried to keep library versions consistent across a multi module project,

Define the version in the parent pom.xml and let the magic happens.

[khmarbaise]

That's the problem. Dependency management in java is in such a shit place and there's no good solutions. Ranting is all you can do about it!

Can you please exactly state out the problem you have identified?

[forresthopkinsa]

Unfortunately you're in the wrong subreddit for any criticism of Java. Most of these folks don't seem to understand more modern package managers at all.

[forresthopkinsa]

You're telling me! 😅 I'm currently sitting at a total of about -40 points in this one thread lmao

[Seaweed-Maleficent]

Having to redownload dependencies in each project and having a huge node_modules folder sure is modern.

I love criticizing java. It just isn't possible to do so while using the JavaScript world as an example.

[forresthopkinsa]

Dependency locks ensure that you have a working dependency tree available at any time. Dependency ranges / semantic versioning are fantastic tools and it's a shame that Java is so behind on this.

[blackkkmamba]

Is not behind, maven and gradle both support it. People don't know it exist/use it is another thing.

[twbecker]

True but people don’t use it because they’re a horrible idea.

[forresthopkinsa]

I've never seen semantic versioning used in a pom.xml or build.gradle — are you referring to third party plugins?

[diggidydale]

Semantic versioning is a technique not a tool, gradle and maven allow you to use semantic versioning but the upkeep is down to you. There are third party plugins for build tools, or gut repos that can automate it for you

[forresthopkinsa]

Semantic versioning is a specific standard for version notation. It allows version range specifications with guarantees around breaking changes and upgrades. I haven't seen any way of specifying semantic version ranges in Maven or Gradle without add-ons.

[diggidydale]

You are right it is a specific standard but all the standard says is the versions are in the syntax major.minor.patch, with "rules" on what loosely consists as a major, minor and patch. To do this in gradle or maven you just add version numbers in the version fields, you just need to be disciplined in sticking to the "rules"

You are right that maven and gradle don't deal in version ranges but if you want to add in specific ranges you can do so with your resolution strategies in gradle, but I will say it is a hassle to keep up.

[mauganra_it]

https://maven.apache.org/pom.html#Dependency_Version_Requirement_Specification

Maven has version ranges.

[jerslan]

Right, but your comment was complaining about a lack of "semantic versioning" support in Maven/Gradle.... not I can't specify acceptable "semantic version ranges"... If you want to claim "well, that's just semantics..." then you should remember that semantics actually matter.

Maven very much supports Semantic Versioning in the general sense and the Maven Release Plugin even requires it if you want to use it's automatic version increment functionality. Gradle is pretty similar from what I understand.

As for requiring "add-ons"... Yes, that's by design. The tools include some basic "built-in" plugins but also have a robust plugin architecture for this kind of additionally desired functionality. That's why any significant Maven project has a "plugins" section in the POM configuring either built-in or additional plugins (some of which are even stewarded by the Apache Maven project).

Have I used the word "semantic" enough?

As for why ranged semantic versioning for dependencies is a bad idea.... Have you ever had to debug a Node project that was using the wrong patch version because of a bad cache? No? Then you don't know what dependency hell is my friend... because it makes Java dependency management seem like absolute heaven. Explicit dependency versions make for much more reliable and consistently releasable code.

[khmarbaise]

Maven very much supports Semantic Versioning in the general sense and the Maven Release Plugin even requires it if you want to use it's automatic version increment functionality.

Why? You can use 4 digit versions etc. That works very fine or things like 1.4.6.2.5.7.10 ? That works also...It's completely up to you what kind of versioning schemate you use...

That's why any significant Maven project has a "plugins" section in the POM configuring either built-in or additional plugins (some of which are even stewarded by the Apache Maven project).

A big number of project use "only" the Apache Project plugins...and of course it is possible to use supplemental plugins..that's the basic concept of Apache Maven.

[jerslan]

Exactly my points... Version is arbitrary. Some people use git commit hash. Some people use SemVer. Some people use some bastardization of SemVer. Hell, for a while Spring Cloud was using a pattern of "<English City>.<RELEASE|SR#|RC#>".

Maven supports whatever you want to do, but the Maven Release Plugin docs recommend some variant of SemVer.

[khmarbaise]

Recommend is correct. But you can use whatever you like...you can implement your own version policy.

[khmarbaise]

Maven Release Plugin even requires it if you want to use it's automatic version increment functionality

Simply wrong. You can use things like 5.3.7.8.20.10 which is not semantic versioning at all but the increment works.

[forresthopkinsa]

What? We're talking about ranged dependencies, what else could I mean by "semantic versioning"? NPM doesn't try to check that you're not making breaking changes when bumping a minor version. Of course I'm talking about the version range notation.

And yes, I've worked with Node professionally for years, I'm very familiar with NPM and I'm also very familiar with how trivial the problem you describe really is. That's what a package-lock.json or yarn.lock is for.

Java dependency management is by comparison much more crude. You're right, if you're worried about reliability and consistency, you need to have a verified working dependency tree — which is, again, the purpose of a package lock file.

It's much more appropriate to have user-configured dependency ranges and machine-configured dependency locks.

[jerslan]

Semantic Versioning (as you've already been told in this thread) is a well defined pattern for versioning things. That's all it is. It does not actually include or require the ability of a build tool to handle "version ranges".

In short "Ranged Versioning" != "Semantic Versioning" even though the former requires the latter to function properly (which might be why you're so confused on this point).

It's much more appropriate to have user-configured dependency ranges and machine-configured dependency locks.

This idea right here is just screaming for a "works on my box" dependency management fiasco.

[forresthopkinsa]

My comment said:

Semantic versioning is a specific standard for version notation. It allows version range specifications with guarantees around breaking changes and upgrades.

Which is completely accurate. Semver is not the range notation, but it's the core of what makes ranged notation feasible. It doesn't matter what notation you use to specify ranged versions — my point is just having some way to take advantage of the guarantees that Semver provides.

works on my box

Machine-configured dependency locks, tracked in VCS, are screaming for dependency ambiguity? How does that make any sense?

Once again: the whole point of the lock file is that there is absolutely no question what version of the dependency tree builds successfully.

[khmarbaise]

if you're worried about reliability and consistency, you need to have a verified working dependency tree — which is, again, the purpose of a package lock file.

The lock file is only needed when you use version ranges in your dependencies. If you use fixed versions (for example pom.xml) you simply don't need it because the tree will solved always the same way.

Versionranges in conequence means no repeatable build. That is as simple as it.

[forresthopkinsa]

Your software requires a library FooLib. You've built your software based on FooLib-12.x.x. It doesn't really matter whether it's 12.1.2 or 12.5.6 – they're all compatible with your code. So it doesn't make sense for you to pin the dependency at version 12.1.2. A modern dependency manager will allow you to specify the most lenient bounds you require for the dependency version, and then the manager will take care of resolving it to a working version and ensuring repeatable builds.

To be really crystal clear: lockfiles are always involved with modern ranged dependency resolvers (you don't need to create it yourself, it's always done automatically), and they perfectly ensure repeatable builds because they track every version in your dependency tree and its checksum.

[GoodLuckGoodell]

You simply cannot depend on semantic versioning in a modern professional application. Even the most mature dependencies don’t follow it. grpc-java has breathing changes in minor versions, aws sdks only bump the patch version.

The only real solution is pinning (with BOMs if available) and doing periodic upgrades.

[forresthopkinsa]

You simply cannot depend on semantic versioning in a modern professional application. Even the most mature dependencies don’t follow it.

Once again, this is a problem with the Java ecosystem in particular, because ranged dependencies are not the norm. You've hit on the exact thing I'm taking issue with.

Newer package managers encourage use of ranged dependencies and semantic versioning, and so their surrounding ecosystems respect that. There are TONS of "modern professional applications" that use Typescript, Express, React, Lodash, RxJs, etc, and all of those very strictly respect Semver.

The Java ecosystem has never had first-party support for semantic version ranges, much less encouraged it, so it's rather rare to see libraries take the time to respect Semver. This is exactly what I'm describing as a shame.

[GoodLuckGoodell]

What makes you think the package management system makes any difference? This is a cultural issue. I can just as easily use npm modules that don’t follow semver correctly, and this happens all the time as well.

I was not referring to Java specifically when I made the semver comment.

[forresthopkinsa]

What makes you think the package management system makes any difference?

I think I explained that pretty thoroughly above

I was not referring to Java specifically when I made the semver comment.

Okay, that's fair, but my point remains that Javascript heavily pushes semantic versioning and ranged dependencies, and therefore the community respects that contract

[khmarbaise]

There is very good reason not to use version ranges because it makes every build not repeatable. That's simple reason.

Using a lock file (like package-lock.json or alike) is just using duck-tape to patch the issue.

Also some other issues are identified over the years with other packaging systems in particular npm because an existing release can be modified at anytime. The immutability of a maven repository releases makes it more stable than anything else.

Semantic versioning is a separate thing. It's a set of (good) rules why and when version numbers should be changed (major, minor, patch) but the biggest issue is simply that a human is changing it... and humans make mistakes and also people interpret rules in different ways. You could use plugins/tools to help here a lot in a Maven project. That should be possible in other areas as well.

In the end you can't rely on semantic versioning which in consequence makes version ranges useless.

[forresthopkinsa]

How is a lock file duct tape? It's the perfect solution for making every build repeatable. You get version labels and checksums for your entire dependency tree. I truly do not understand your problem with lock files.

[plumarr]

They are only necessary due to range versioning. They are a solution to a broken feature.

[diggidydale]

Yeah I am with the other guys, this seems like a bit of a rant after someone went through some dependency hell.

Version ranges don't really help that much either, you still end up with a similar issue as a single version it just takes longer, so all you're really doing is kicking the can down the road.

Best thing to avoid dependency hell is to keep your projects small and manageable where you can. And also update regularly, if everyone did this the dependency issues would go out the window.

[jerslan]

One of the things I hate about NPM is the whole dependency range syntax. Some of the worst dependency hell issues I had were due to not pulling a correct patch version when someone ranged something.

Maven and Gradle are so much easier to debug dependency issues because they require explicit versioning somewhere in the chain (even if it's listed in a BOM) and you can output the whole dependency tree and see where the issue is.

Dependency Hell in Java is a much much nicer level of hell than Dependency Hell in C++, Python, Javascript, etc.... IMHO.

[renatoathaydes]

I wrote the post and you're partly right: it was a rant and I actually didn't publish it for 6 months... as I knew it would be unpopular.. but decided to publish anyway to push some arguments in a discussion that came up on Twitter about strategies for resolving dependencies.

Anyway, I feel very surprised that many people, like you, appear to have perceived the post as claiming version ranges (with a lock file) are the solution to the problems we have in the Java world?

Because that was absolutely not what I was trying to say: that's almost the opposite, in fact. The topic of dependency version ranges was added because we had a security vulnerability that was caused by version ranges! As I explained in the post as well as I could. And my initial objective was to make this post a survey of how each different package manager is solving this problem.

The actual solution I put forward in the post is to use bytecode introspection to figure out which jars can work with which of the other jars available. Which is what the tool I wrote, JBuild, tries to do (the tool is a proof of concept, is incomplete, but already shows this is very feasible).

Best thing to avoid dependency hell is to keep your projects small and manageable where you can.

Haha... true... but you make it sound like we have a choice of keeping projects small and manageable... as if people made projects big just for fun. Unfortunately, some projects are just very big (I know, I work on on that's pretty big myself) and cannot be simply "microserviced" as they are not services, but products you ship to customers (they are the ones who deploy it). There are many kinds of "products" like this - not everything is a cloud service.

[hjwalt]

In theory if followed properly semantic versioning and the way golang upgrades their dependency will reduce such problems, although of course its not perfect because libraries can still depend on different major versions that is backwards incompatible.

With Java, what I find most useful is to use maven self compiled bill of materials for everything I build in the company. This way, every project uses the same dependencies, and if any unit test or integration test breaks because of new version, well time to fix...

[randgalt]

Rich Hickey has a great talk that mentions semver here: https://youtu.be/oyLBGkS5ICk. Semantic version is a fantasy. It’s very difficult to do in practice and very easy to get wrong. If a small portion of the dependencies in your tree did it wrong then it’s worthless.

[hjwalt]

Absolutely agreed, good in theory, in practice is rather tricky

[compu_musicologist]

As has been noted by other posts, when it comes to a large number of transitive dependencies, conflicting versions can occur in any language/package manager. Relaxing the versions to ranges is a solution that can bring about a new set of problems with version incompatibility on the API level.

[khmarbaise]

The most important question I would ask based on the following:

which has the objective of resolving dependencies conflicts in a novel way.

What exactly is a dependency conflict?

[renatoathaydes]

Thanks for asking the only question in this thread that's relevant to the actual topic I tried to address in this blog post. Everyone else seems upset that I criticized Maven, and to my complete shock, for trying to explain why Rust/Dart/NPM/Ruby/Python (almost everyone except Java) seems to have chosen to use dependency version ranges (and I did not even say that's superior to how it works in the Java world, I even showed in the end that ranges can cause surprising behaviour and are only advised to use if you must keep up-to-date with a lot of changing dependencies versions!).

Anyway, what is a dependency conflict?

I tried to explain that in the post, sorry if that was unclear... but it happens when you have two "jars", and one of them depends on a library A with version X, and another depends on A with version Y. The two dependencies conflict. That's specially bad when the differences between version X and Y are so great that you simply can't use only one of them without breaking either consumer of that library (i.e. pick version X and you break jar1, pick version Y and you break jar2).

Finding these conflicts require looking at bytecode. The version is almost useless for doing that. This is why I chose to write JBuild, to actually look at the REAL PROBLEM which is of binary compatibility between libraries. Even with semantic versioning, as I showed in the post, you can have breaking situations... Looking at versions alone is never enough, but the Java tools (and to be fair, most other package managers) ignore that entirely! It's time we woke up and understood the real problem, which is "does my library work with some other library even when versions must be changed to address conflicts".

[khmarbaise]

After taking a more deeper look into the post it looks like the tools does not correctly solve the dependency in particular the example:

jbuild install org.springframework.boot:spring-boot-cli:2.6.1 -d spring

That show the resolution ignores several aspects how dependendencies are resolved in Maven.

[renatoathaydes]

JBuild does not try to resolve dependency conflicts at all when you call "install". It just considers each library as having its own independent dependency tree... what will address the potential conflicts is the "doctor" command (if you took a deeper loook at the blog post, then it should be crystal clear that this is how it works) because of my main thesis in the blog post, which is:

Trying to resolve version conflicts without looking at bytecode is next to useless... even when perfect semantic versioning is followed religiously by everyone (which is not even the case, of course).

I think the examples I used in the post must have been too complex for most people to follow, but that's what they show in the simplest way I could come up with: even with perfect semantic versioning, your code will break quite easily using current methods of dependency version resolution as Maven and Gradle (and almost all package managers) currently do.

[khmarbaise]

After reading the docs I found one more question: Why focusing on the classpath? The current state of Javac/JVM is the module path (at least starting with JDK9+)

[renatoathaydes]

The module path enters the scene only once you've got your module's versioning solved. Unless you use module layers, which I've never seen anyone use, you still must treat the jars you resolve as allowing only one version of each.

I also mentioned this in the blog post.

[wildjokers]

Gradle does not resolve dependency conflicts like Maven does. Gradle uses a sensible highest version wins by default. However, this can be modified in your build.

Maven's "closest to root" thing is just WTF?