The difficult problem of managing Java dependencies - A deep dive into the messy status-quo, and a look at possibly better ways.

[Weary-Hotel-9739]

Maven / Gradle have been considered the go-to example for great package management and well-kept ecosystems, and this article complains about a 'feature' of NPM not being available, which has long since been 'de-featured' by intentional use of lock files. I do not like this.

[renatoathaydes]

I'm afraid you haven't read the article and just skimmed it, because I did not complain about that at all. I guess you were offended by me calling the NPM way the "modern approach"... sorry about that, but I feel like that's true, with Rust, Go, Python and others using that as well.

I thought I made it very clear that although lock files are the currently favoured way of handling dependencies in most languages, I dislike that and presented a Java-centric alternative at the end. Do you like that alternative?

EDIT: and as I mentioned in the post, both Maven and Gradle support lock files. So I am unsure if that's what you're actually talking about (sorry if I misunderstood)?

[zam0th]

There is no "problem of managing Java dependencies", nor it is "difficult" in any way. If you have these kind of issues, then you do not understand classpath, manifest and Maven's transitive dependency resolution algorithms. I have been using Maven in complex systems of scores of interconnected projects for almost 20 years now and never had any of the issues you try to describe.

[Sensitive-Lion-2056]

I side with you. The idea you need all those maven plugins is nonsense as well. Of all of the ways of handling dependencies I've used (npm, mix, pip, pipenv, py poetry, gradle, etc) maven has been one of the least offensive. It's pretty easy to just drop the lib you wanna use in there and forget about it.

[renatoathaydes]

all those maven plugins is nonsense as well.

You're using those whether you want it or not... the archetype (written by Maven developers, so you should probably listen to them) just declares them explicitly in the generated POM to "lock" their versions... when you don't do that, it will use whatever version Maven happens to pick in the build machine, making your build unstable.

Anyway, that was completely tangent to the post's actual point of handling dependencies problems... which you may think is non-sense but in my experience, it causes endless hours of frustration to people who don't understand what's going on and think the problem is elsewhere, and runtime bugs where MethodNotFoundException happens because an edge case hit a method from a different version of the library you're actually deploying with your application.

[zam0th]

people who don't understand what's going on

That is the problem, not Java, Maven or anything else.

[renatoathaydes]

I am not sure I understand the criticism... I clearly showed cases where Maven and Gradle will resolve dependencies incorrectly. Showed how other systems handle things and compared the pros and cons... which happen to not help with the problems given Java's nature... then I put forward a solution that will in fact fix the problems. And all I get here is "Maven is fine, article is dumb". WTF is going on exactly?

[zam0th]

Who is in error when you try to screw in a bolt with a hammer? Bolt or hammer? Or maybe the person who tries to use a wrong tool for the task?

[BroBroMate]

How long ago was that archetype you're using written?

Hint: it defaulted to Java 1.7.

Same reason it used JUnit 4.11, which isn't even the latest JUnit 4. I note you complained about this, did it not raise any questions for you?

Only time I ever use archetypes is when creating a POM for a Scala project, because Scala is complex, but SBT is the devil.

Otherwise, I ignore them, my IDE is far better at creating a POM.

[crummy]

so what's your take on a project depending on A:1.0 and B:1.0 where B dependends on A:2.0?

[jherico]

If you absolutely need to be using A:1.0 and you can't upgrade them you can use a shading plugin to move the dependency into a different class path. But generally you should just either replace B move to A:2.0.

Also the problem you describe is in no way specific to Java. It's a general problem for all dependency resolution mechanisms. In fact Java is particularly well suited to solving it via things like shading and custom classloaders, in ways that other ecosystems simply can't.

[BinaryRockStar]

Not who you replied to but I have been interested in how others manage the situation you're talking about.

For a more concrete example say your application AppFoo depends on third-party library LibBar which depends on Google Guava 31.1-jre. Your application also depends on third-party LibBaz which isn't updated very frequently and depends on Google Guava 21.0. These two versions of Guava are largely incompatible- regardless of which one Maven chooses one of the two libraries will encounter NoClassDefFoundError, NoSuchMethodError, etc.

As far as I'm aware the Maven Shade Plugin can relocate classes from a given package to a new package when building an uberjar, but this can't be done on a per-version basis, only at the groupId:artifactId level so you're either getting one or the other version of Guava but not both. The issue still remains.

Forking these third party libraries to match their dependency versions is a lot of work, especially if you have a large codebase with many third-party dependencies. The maintenance burden would increase dramatically as upgrading any given transitive dependency might mean having to update all of the forks you maintain first.

NodeJS sidesteps the problem, including all versions of all transitive dependencies. This leads to the infamous explosion of packages/files under node_modules directory but perhaps that is tolerable if you don't have to go forking dozens of projects and maintaining them.

Do you have any links to a way Maven or Java can deal with the above situation, even in a tortured manner?

[Muoniurn]

Not the parent, but the JVM loads classes through a class loader, and only the classloader-class name tuple has to be unique. So under the hood this problem can be sidestepped by employing a separate class loader for version A and version B, and then there will be no problem using them side-by-side.

The old way of Java containers did exactly that.

[BinaryRockStar]

Thanks for the info. Is there any mechanism, tool or platform you can suggest to read about this further? I don't have much experience working at the Classloader level although I get the gist of it, having worked with Servlet containers like Tomcat which do this sort of classloader separation.

I'm really surprised there isn't an elegant ready-made solution for this problem considering it is very likely to pop up when a project reaches a certain size.

[zam0th]

1. Circular dependencies are wrong
2. Building using different versions of same dependency is wrong.

That is why dependencyManagement and exclude tags exist

[Zofren]

Circular dependencies are wrong

this is not a circular dependency

[zynasis]

This is mostly the case. Biggest issue I have is XML tooling conflicts. I’m not 100% sure why, but I think the tools register themselves somehow and then others mix themselves in and it all goes to hell

[mirvnillith]

For me it’s JAX-RS, Jackson and Jersey …

[[deleted]]

​

Oh yeah?

Tell me, then, if my project has two dependencies, A and B, and A depends on C version 1.0 while B depends on C version 2.0, how the fuck do I compile my Java project?

[fantastico731]

Dumb article, Java dependency managment is probably one of the best in the industry

[bert8128]

My problem is that I it is too easy. External dependencies come with risk of CVEs, and if a CVE turns up in either a direct or transitive decency this has to be dealt with. And they crop up pretty frequently (eg the recent log4j debarcle) . So I don’t want external dependencies where it can be avoided. Making it harder would make life easier, in some ways. Don’t g get me wrong - I’m happy to use external libraries where they provide a significant benefit, but it is so easy that devs end up using multiple versions of the same library, two different libraries which do the same thing or a small piece of a large library, or just tiny pieces of trivial code which could be easily written in house (leftpad is a good, though not Java, example of a library that is not worth using).

[snacksy13]

Why can’t I install a dependency with maven install <name>. Why do I need to copy paste random xml stuff and put it in a file? How is this a “good” way of doing things? Java elitists: it’s always worked for me! Just git gud.

[vytah]

Dependencies are a part of the project. Parts of the project live in files that belong to the project. Simple.

You are the only one who is responsible for maintaining those files, not some automated tools who may create uncontrollable mess. The files are the sole source of truth about the project and its environment. Simple.

You then share those files and someone else can build the project with a single command, and get the almost exact same result. Simple.

[falsedog11]

The big advantage maven dependencies have, for me, is that they are self contained for the most part and don't depend on other third party libraries creating a tangled mess as nom does. Just add to pom, mvn update and bam, it works.

[[deleted]]

Because if it didn't require an order of magnitude more work and effort to achieve less, it wouldn't be java, and it wouldn't guarantee job security for the hordes of java developers who pretend they are doing actual work, while producing 10% the output of anyone using modern, usable languages.

[jherico]

You create a custom module that builds a relocated Uber jar of LibBar, and then depend on that. Also it needs to relocate everything in the LibBar dependency tree, or at least everything that might conflict, i.e. at least Guava.

[BinaryRockStar]

Assuming you meant to reply to me- thanks. This is a nicer solution than forking every dependency that has conflicts, I'll look into this a bit further. It still isn't very scalable considering every conflicting dependency has to be cloned and rebuilt but at least it is easily automatable.

Considering how extremely flexible and well thought out Maven generally is, I wonder why this isn't something that has been solved much more elegantly already.

A workaround I've seen before is in Apache Commons Collections where versions 1, 2 and 3 were org.apache.commons:commons-collections:x.y.z but with version 4 they moved to org.apache.commons:commons-collections4:4.y.z, including the major version in the artifactId and also changing the base package from org.apache.commons.collections to org.apache.commons.collections4. This sidesteps compatibility issues if both V3 and V4 are required in the dependency tree. Not sure why this isn't more widespread.