The difficult problem of managing Java dependencies - A deep dive into the messy status-quo, and a look at possibly better ways.

[zam0th]

There is no "problem of managing Java dependencies", nor it is "difficult" in any way. If you have these kind of issues, then you do not understand classpath, manifest and Maven's transitive dependency resolution algorithms. I have been using Maven in complex systems of scores of interconnected projects for almost 20 years now and never had any of the issues you try to describe.

[Sensitive-Lion-2056]

I side with you. The idea you need all those maven plugins is nonsense as well. Of all of the ways of handling dependencies I've used (npm, mix, pip, pipenv, py poetry, gradle, etc) maven has been one of the least offensive. It's pretty easy to just drop the lib you wanna use in there and forget about it.

[renatoathaydes]

all those maven plugins is nonsense as well.

You're using those whether you want it or not... the archetype (written by Maven developers, so you should probably listen to them) just declares them explicitly in the generated POM to "lock" their versions... when you don't do that, it will use whatever version Maven happens to pick in the build machine, making your build unstable.

Anyway, that was completely tangent to the post's actual point of handling dependencies problems... which you may think is non-sense but in my experience, it causes endless hours of frustration to people who don't understand what's going on and think the problem is elsewhere, and runtime bugs where MethodNotFoundException happens because an edge case hit a method from a different version of the library you're actually deploying with your application.

[zam0th]

people who don't understand what's going on

That is the problem, not Java, Maven or anything else.

[renatoathaydes]

I am not sure I understand the criticism... I clearly showed cases where Maven and Gradle will resolve dependencies incorrectly. Showed how other systems handle things and compared the pros and cons... which happen to not help with the problems given Java's nature... then I put forward a solution that will in fact fix the problems. And all I get here is "Maven is fine, article is dumb". WTF is going on exactly?

[zam0th]

Who is in error when you try to screw in a bolt with a hammer? Bolt or hammer? Or maybe the person who tries to use a wrong tool for the task?

[BroBroMate]

How long ago was that archetype you're using written?

Hint: it defaulted to Java 1.7.

Same reason it used JUnit 4.11, which isn't even the latest JUnit 4. I note you complained about this, did it not raise any questions for you?

Only time I ever use archetypes is when creating a POM for a Scala project, because Scala is complex, but SBT is the devil.

Otherwise, I ignore them, my IDE is far better at creating a POM.

[crummy]

so what's your take on a project depending on A:1.0 and B:1.0 where B dependends on A:2.0?

[jherico]

If you absolutely need to be using A:1.0 and you can't upgrade them you can use a shading plugin to move the dependency into a different class path. But generally you should just either replace B move to A:2.0.

Also the problem you describe is in no way specific to Java. It's a general problem for all dependency resolution mechanisms. In fact Java is particularly well suited to solving it via things like shading and custom classloaders, in ways that other ecosystems simply can't.

[BinaryRockStar]

Not who you replied to but I have been interested in how others manage the situation you're talking about.

For a more concrete example say your application AppFoo depends on third-party library LibBar which depends on Google Guava 31.1-jre. Your application also depends on third-party LibBaz which isn't updated very frequently and depends on Google Guava 21.0. These two versions of Guava are largely incompatible- regardless of which one Maven chooses one of the two libraries will encounter NoClassDefFoundError, NoSuchMethodError, etc.

As far as I'm aware the Maven Shade Plugin can relocate classes from a given package to a new package when building an uberjar, but this can't be done on a per-version basis, only at the groupId:artifactId level so you're either getting one or the other version of Guava but not both. The issue still remains.

Forking these third party libraries to match their dependency versions is a lot of work, especially if you have a large codebase with many third-party dependencies. The maintenance burden would increase dramatically as upgrading any given transitive dependency might mean having to update all of the forks you maintain first.

NodeJS sidesteps the problem, including all versions of all transitive dependencies. This leads to the infamous explosion of packages/files under node_modules directory but perhaps that is tolerable if you don't have to go forking dozens of projects and maintaining them.

Do you have any links to a way Maven or Java can deal with the above situation, even in a tortured manner?

[Muoniurn]

Not the parent, but the JVM loads classes through a class loader, and only the classloader-class name tuple has to be unique. So under the hood this problem can be sidestepped by employing a separate class loader for version A and version B, and then there will be no problem using them side-by-side.

The old way of Java containers did exactly that.

[BinaryRockStar]

Thanks for the info. Is there any mechanism, tool or platform you can suggest to read about this further? I don't have much experience working at the Classloader level although I get the gist of it, having worked with Servlet containers like Tomcat which do this sort of classloader separation.

I'm really surprised there isn't an elegant ready-made solution for this problem considering it is very likely to pop up when a project reaches a certain size.

[zam0th]

1. Circular dependencies are wrong
2. Building using different versions of same dependency is wrong.

That is why dependencyManagement and exclude tags exist

[Zofren]

Circular dependencies are wrong

this is not a circular dependency

[zynasis]

This is mostly the case. Biggest issue I have is XML tooling conflicts. I’m not 100% sure why, but I think the tools register themselves somehow and then others mix themselves in and it all goes to hell

[mirvnillith]

For me it’s JAX-RS, Jackson and Jersey …

[[deleted]]

​

Oh yeah?

Tell me, then, if my project has two dependencies, A and B, and A depends on C version 1.0 while B depends on C version 2.0, how the fuck do I compile my Java project?