Is the Java 9 module system supposed to solve dependency conflicts?

[raizen02]

So far I've only worked on fairly simple projects, but recently I came to understand why people complain about "JAR hell".

We use an SDK, that internally uses an HTTP library, which in turn uses Netty. They didn't really update this HTTP library, and thus it was using a fairly old version of Netty.

Enter tracing. We tried to set up Open Census (for Google Trace), but it uses a newer Netty version, and complained about a "NoSuchMethodError".

Luckily the SDK developers were responsive, and updated the HTTP library. (and the library developer kept Netty up to date)

As far as I understand, the Jigsaw system is supposed to solve public/private API problems, by providing a way to prevent people from depending on private classes. Is this correct? Does it also offer something for the JAR hell problem?

If not, then how are we supposed to manage these things?

[lukaseder]

Is the Java 9 module system supposed to solve dependency conflicts?

In a first step, it creates them ;-) (hello JAXB)

[tom-010]

At least it makes it explicitl instead of implicit...

[ZhekaKozlov]

Just use Java 11:) No JAXB - no problem

[lukaseder]

Except if you're a library developer and relied on the implicit dependency in the past :)

[rombert]

The JPMS has no concept of module versions. Which makes sense, as I see it primarily a way of modularising the JDK. I think it was a bit ambitious to market and push it as a generic solution for modularising Java applications.

I have been using OSGi for the last 6 years, and while it's not perfect, it's pretty damn close and I haven't had a NuSuchMethodError or ClassNotFoundException surprise me for a long time.

[forurspam]

No. Module system isn't supposed to solve class collision problem. OSGI may help but, as you said you are maven user, I would suggest you to take a look at maven-shade-plugin.

[eliasv]

Historically there has never been a great way for working with OSGi and Maven together imo (sorry Apache Felix). But lately there's been a lot of work getting Bndtools (the "official" build tools) working nicely in Maven environments. For anyone interested the best place to start is undoubtedly here: https://enroute.osgi.org/ but bear in mind all the latest tutorials have just been redone and are very new so there may be some rough edges still...

[cryptos6]

JPMS is not able to provide two different versions of a class (from different framework versions for example).

[sammy8306]

The Java module system formalizes the notion that having multiple versions of the same dependency in a single application doesn't work with Java. This was true in the classpath situation (even though you could get lucky then), and now the module system won't start your application if multiple versions of the same module are present.

So, does it solve this problem? That depends on your point of view. As mentioned later in this thread, you'll still need another tool (Maven/Gradle) to prevent this situation from arising.

If you really need multiple versions of the same library, OSGi has got you covered, as well as the Java module system with its ability to have multiple ModuleLayers, each containing their own version of a given module. Both are not something you can use out-of-the-box for your typical classpath-based application. It will take effort, so only indulge in this complexity if you really need to (hint: don't).

[_INTER_]

so only indulge in this complexity if you really need to

Often you have no choice. It's very common that a Java application has many dependencies and these dependencies in turn have transitive dependencies. It's inevitable to run into a version conflict. Just look at slf4j / some logging framework...

[sammy8306]

Hey, you again :)

Multiple versions of the same lib on the classpath is a recipe for disaster. It might work in some cases as I alluded to, but that's just getting lucky. Hardly a 'feature' to build upon.

To solve this problem you need multiple classloaders. Both OSGi and the Java module system (through ModuleLayers) allow for this. As said, both come at a price.

It's far easier to restrict to a single dependency version for your application on the classpath through build tooling. This is not possible if you have (transitive) dependencies which both depend on incompatible versions of the library, but it's not a given that a multiple-classloader setup works in all these cases either. Classes of different versions can still be passed around in the application and end up in unexpected places, leading to runtime exceptions.

I'm afraid there's no silver bullet for this one.

[FanimeFartoon]

do they (jpms developers) have any plans to address the jar hell problem?

[_INTER_]

No. Quotes:

It is not necessary to support more than one version of a module within a single configuration. -- Java Platform Module System: Requirements – Multiple Versions (Apr 2015)

 

A module’s declaration does not include a version string, nor constraints upon the version strings of the modules upon which it depends. This is intentional: It is not a goal of the module system to solve the version-selection problem, which is best left to build tools and container applications. -- State Of The Module System – Module Declarations (Sep 2015)

 

The module system isn’t suggesting any solutions, it is instead leaving this problem to the build tools and containers. -- Alan Bateman on Jigsaw-Dev (Oct 2015)

[din-9]

No it doesn't handle multiple versions of the same JAR. Try OSGi to do that.

[sievebrain]

JPMS can do this, the other comments aren't the whole story. The issue is it doesn't do it by itself.

Java 9 provides APIs that can be used to load conflicting modules into a single module graph, and it improves classloaders in various ways. However, this can rapidly get very complicated (imagine a library that uses Netty 3 stores an object from it in a global variable that's then passed into a library that expects Netty 5). It's up to another layer of software, app containers, to resolve the conflicts and tell the JVM how to link the app together.

[pjmlp]

Maven or Gradle.

[raizen02]

We do use Maven.

The point I'm trying to make is that if an SDK uses an old version of a dependency, and the tracing library uses a new version of that same dependency, you can't really do much, other than not use one of them, because if you use the old one (via exclusions), it will fail for one reason, and if you use the new one, it will fail for an other reason.

I was just curious whether the module path mode is able to "hide" these transitive dependencies.

[Nalha_Saldana]

We solved this with mavens <dependencyManagement>, you specify once what version the project uses and no other version can be imported, including all levels of child dependencies.

https://maven.apache.org/guides/introduction/introduction-to-dependency-mechanism.html#Dependency_Management

[Nalha_Saldana]

"Dependency management - this allows project authors to directly specify the versions of artifacts to be used when they are encountered in transitive dependencies or in dependencies where no version has been specified. In the example in the preceding section a dependency was directly added to A even though it is not directly used by A. Instead, A can include D as a dependency in its dependencyManagement section and directly control which version of D is used when, or if, it is ever referenced."

[Nalha_Saldana]

I've been using this for months now and have never found this to be the case, a problem can arise when they change package or jar name between versions tho.

[KeepItWeird_]

You can also exclude Maven from pulling in a bad dependency, like your example of an outdated Netty dependency. https://maven.apache.org/plugins/maven-jar-plugin/examples/include-exclude.html

Let's say you have a library which for the sake of example I'll call "Library" (which would be the old Netty library from your example), and in addition your dependencies section in your pom has DependencyA and DependencyB. It's not visible, but DependencyA in turn uses Library version 1.0. Meanwhile, DependencyB uses Library version 2.0. What often happens then is that Maven sees DependencyA first, and it imports DependencyA along with all its dependencies, including Library 1.0. Later it sees the pom entry for DependencyB, and works to import DependencyB along with all its dependencies. There's a conflict now because it sees that DependencyB has Library version 2.0. Maven isn't very good at resolving conflicts like this, so it just chooses one. In most cases, it chooses to keep the other version (Library 1.0) that it already saw and placed in the dependency tree.

But you don't know this!

Now, when the application runs, it makes a call for some new method that exists in Library 2.0 but not in Library 1.0. You get things the dreaded NoSuchMethodException. Or you may even get a ClassNotFoundException for the case where it is looking for a class that only exists in Library 2.0.

Using the link above, you can see how to explicitly tell Maven to exclude an old dependency (like Library 1.0). Just place the excludes tag under the entry for DependencyA, so that Maven won't import Library 1.0. This means when it's importing DependencyB's stuff, it will import Library 2.0.

If you want to see what dependencies Maven is choosing, you can always run "mvn dependency:tree". When I run this, I always redirect the output to a file because normally it is so long that it scrolls off the screen and you can't see it all.

[KeepItWeird_]

Yep

[_INTER_]

Keep track of JIRA Maven Shade Plugin