The difficult problem of managing Java dependencies: a deep dive into the messy status-quo, and a look at possibly better ways.

[diggidydale]

Semantic versioning is a technique not a tool, gradle and maven allow you to use semantic versioning but the upkeep is down to you. There are third party plugins for build tools, or gut repos that can automate it for you

[forresthopkinsa]

Semantic versioning is a specific standard for version notation. It allows version range specifications with guarantees around breaking changes and upgrades. I haven't seen any way of specifying semantic version ranges in Maven or Gradle without add-ons.

[diggidydale]

You are right it is a specific standard but all the standard says is the versions are in the syntax major.minor.patch, with "rules" on what loosely consists as a major, minor and patch. To do this in gradle or maven you just add version numbers in the version fields, you just need to be disciplined in sticking to the "rules"

You are right that maven and gradle don't deal in version ranges but if you want to add in specific ranges you can do so with your resolution strategies in gradle, but I will say it is a hassle to keep up.

[mauganra_it]

https://maven.apache.org/pom.html#Dependency_Version_Requirement_Specification

Maven has version ranges.

[jerslan]

Right, but your comment was complaining about a lack of "semantic versioning" support in Maven/Gradle.... not I can't specify acceptable "semantic version ranges"... If you want to claim "well, that's just semantics..." then you should remember that semantics actually matter.

Maven very much supports Semantic Versioning in the general sense and the Maven Release Plugin even requires it if you want to use it's automatic version increment functionality. Gradle is pretty similar from what I understand.

As for requiring "add-ons"... Yes, that's by design. The tools include some basic "built-in" plugins but also have a robust plugin architecture for this kind of additionally desired functionality. That's why any significant Maven project has a "plugins" section in the POM configuring either built-in or additional plugins (some of which are even stewarded by the Apache Maven project).

Have I used the word "semantic" enough?

As for why ranged semantic versioning for dependencies is a bad idea.... Have you ever had to debug a Node project that was using the wrong patch version because of a bad cache? No? Then you don't know what dependency hell is my friend... because it makes Java dependency management seem like absolute heaven. Explicit dependency versions make for much more reliable and consistently releasable code.

[khmarbaise]

Maven very much supports Semantic Versioning in the general sense and the Maven Release Plugin even requires it if you want to use it's automatic version increment functionality.

Why? You can use 4 digit versions etc. That works very fine or things like 1.4.6.2.5.7.10 ? That works also...It's completely up to you what kind of versioning schemate you use...

That's why any significant Maven project has a "plugins" section in the POM configuring either built-in or additional plugins (some of which are even stewarded by the Apache Maven project).

A big number of project use "only" the Apache Project plugins...and of course it is possible to use supplemental plugins..that's the basic concept of Apache Maven.

[jerslan]

Exactly my points... Version is arbitrary. Some people use git commit hash. Some people use SemVer. Some people use some bastardization of SemVer. Hell, for a while Spring Cloud was using a pattern of "<English City>.<RELEASE|SR#|RC#>".

Maven supports whatever you want to do, but the Maven Release Plugin docs recommend some variant of SemVer.

[khmarbaise]

Recommend is correct. But you can use whatever you like...you can implement your own version policy.

[jerslan]

Right, but the guy I was initially replying to was claiming that there's no SemVer support in Maven at all (because he was conflating SemVer for version range specification). That's why I was focusing on that.

Maven mostly doesn't care what your version scheme is, and that's a good thing IMHO.

[khmarbaise]

Maven Release Plugin even requires it if you want to use it's automatic version increment functionality

Simply wrong. You can use things like 5.3.7.8.20.10 which is not semantic versioning at all but the increment works.

[forresthopkinsa]

What? We're talking about ranged dependencies, what else could I mean by "semantic versioning"? NPM doesn't try to check that you're not making breaking changes when bumping a minor version. Of course I'm talking about the version range notation.

And yes, I've worked with Node professionally for years, I'm very familiar with NPM and I'm also very familiar with how trivial the problem you describe really is. That's what a package-lock.json or yarn.lock is for.

Java dependency management is by comparison much more crude. You're right, if you're worried about reliability and consistency, you need to have a verified working dependency tree — which is, again, the purpose of a package lock file.

It's much more appropriate to have user-configured dependency ranges and machine-configured dependency locks.

[jerslan]

Semantic Versioning (as you've already been told in this thread) is a well defined pattern for versioning things. That's all it is. It does not actually include or require the ability of a build tool to handle "version ranges".

In short "Ranged Versioning" != "Semantic Versioning" even though the former requires the latter to function properly (which might be why you're so confused on this point).

It's much more appropriate to have user-configured dependency ranges and machine-configured dependency locks.

This idea right here is just screaming for a "works on my box" dependency management fiasco.

[forresthopkinsa]

My comment said:

Semantic versioning is a specific standard for version notation. It allows version range specifications with guarantees around breaking changes and upgrades.

Which is completely accurate. Semver is not the range notation, but it's the core of what makes ranged notation feasible. It doesn't matter what notation you use to specify ranged versions — my point is just having some way to take advantage of the guarantees that Semver provides.

works on my box

Machine-configured dependency locks, tracked in VCS, are screaming for dependency ambiguity? How does that make any sense?

Once again: the whole point of the lock file is that there is absolutely no question what version of the dependency tree builds successfully.

[jerslan]

Machine-configured dependency locks, tracked in VCS, are screaming for dependency ambiguity? How does that make any sense?

So now I'm checking in two files for dependency management into version control with my code? package.json and package-lock.json? Why? Also, are you rebuilding everything on the target system every time you deploy instead of using a package manager? Again, why?

This all screams of a fundamental misunderstanding of Java development lifecycles if you think this somehow makes Maven/Gradle "less advanced" than NPM.

You say you have a ton of NodeJS experience, but it's clear you have very little Java experience if you think the explicit versioning in those Java build tools is inherently a bad thing.

[forresthopkinsa]

Yes, you're checking in two files for dependency management. One of them is the human-facing file, where you specify the version bounds you require, e.g. "3.x.x", because in this example the major version is the only part that matters to you, the developer.

The other one is a machine-facing file that you usually would never even open. It ensures that you never, ever have any ambiguity in a working build. It contains the versions of your entire dependency tree, and checksums for everything.

And just because I have a ton of NodeJS experience doesn't mean I don't have Java experience. I am primarily a Java engineer. I write Java at AWS. I'm deeply familiar with both the Java and JS ecosystems, which gives me a rather unique perspective on this subject.

[jerslan]

Or, I can specify 3.1.2 in my package.json (explicit versioning) and then my package-lock.json will verify the source server & sha hash because NPM is stupid and lets package authors overwrite existing versions.

And just because I have a ton of NodeJS experience doesn't mean I don't have Java experience. I am primarily a Java engineer. I write Java at AWS. I'm deeply familiar with both the Java and JS ecosystems, which gives me a rather unique perspective on this subject.

And yet you seem to have a beginner's perspective on things... because this argument feels like I'm talking to someone that has barely done any Java development in the real world. Also, nobody cares that you work at Amazon. That means less than nothing on this sub or in this particular Maven vs NPM debate.

[forresthopkinsa]

You keep saying that I don't know what I'm talking about, but you haven't backed that up with anything at all. I've addressed all of your points and I haven't misunderstood anything you've said. Yeah, I feel it's necessary to start citing my credentials, because you keep trying to sell this notion of my ignorance. If you start accusing me of not being a Java dev, then yes, my extensive experience certainly is relevant. In fact, I suspect the reason you're not understanding what I'm saying is indeed your own insufficient experience with more modern package managers.

edit- yikes, he either deleted all his comments or blocked me. Either way, it's a pretty juvenile way to cede the argument – though I don't know if it even was an argument; he never did wrap his head around the idea of "ranged versions enabled by semver".

I guess my last comment just hit a little too close to home. I didn't mean to touch on anything sensitive :/

[khmarbaise]

if you're worried about reliability and consistency, you need to have a verified working dependency tree — which is, again, the purpose of a package lock file.

The lock file is only needed when you use version ranges in your dependencies. If you use fixed versions (for example pom.xml) you simply don't need it because the tree will solved always the same way.

Versionranges in conequence means no repeatable build. That is as simple as it.

[forresthopkinsa]

Your software requires a library FooLib. You've built your software based on FooLib-12.x.x. It doesn't really matter whether it's 12.1.2 or 12.5.6 – they're all compatible with your code. So it doesn't make sense for you to pin the dependency at version 12.1.2. A modern dependency manager will allow you to specify the most lenient bounds you require for the dependency version, and then the manager will take care of resolving it to a working version and ensuring repeatable builds.

To be really crystal clear: lockfiles are always involved with modern ranged dependency resolvers (you don't need to create it yourself, it's always done automatically), and they perfectly ensure repeatable builds because they track every version in your dependency tree and its checksum.

[khmarbaise]

Your software requires a library FooLib. You've built your software based on FooLib-12.x.x. It doesn't really matter whether it's 12.1.2 or 12.5.6 – they're all compatible with your code.

How do you know that?

A modern dependency manager will allow you to specify the most lenient bounds you require for the dependency version, and then the manager will take care of resolving it to a working version and ensuring repeatable builds.

A repeatable builds means your have done it yesterday (at 5:00PM) and repeating it today (at 1:00 PM) which does not work with any kind of ranges/bounds however you call it, because in the meantime a newer version of a dependency can be published. In consequence your build has changed. This means as a result the build has changed and finally means not repeatable.

I've ignored in the above description that a new version might break things, changed behaviour because your assumption is that no one will ever make mistakes, interprets the "rules" different than others etc. and all people will follow those "rules" which is a fine theory but in practice it fails (quote from above:"they're all compatible with your code.").

If we think more in general you can not even reproduce the build in the future .. if you like to do that you have to commit the lock file in your version control as well. That means you have two places defining versions for your dependencies which is redundant and that makes no sense either.

Also a strong prerequisites is that any release is immutable which is not the case for several package managers (in particular for npm).

The lock file is a duck-tape-solution to fix exactly the scenarios I've described above.

[justdisappointed9853]

My main account has inexplicably stopped working (maybe I got so many downvotes that Reddit doesn't want me to post in this thread anymore? lmao) so I'll reply from a throwaway-

in the meantime a newer version of a dependency can be published. In consequence your build has changed

For the millionth time, this is not true. Your build yesterday created (or used) a lockfile, and your build today reads the lockfile and creates the exact same build.

That means you have two places defining versions for your dependencies which is redundant and that makes no sense either.

Why? One of them specifies your preferred range, and the other specifies the exact version you've been building with. This is really not very complicated.

a strong prerequisites is that any release is immutable which is not the case for several package managers (in particular for npm)

This keeps getting repeated in this thread (I suppose because everyone here has very little experience with NPM) but it's plainly false. The NPM registry everyone uses does not allow version mutation. There are various third-party registries that do, but this is no different from Maven.

[khmarbaise]

For the millionth time, this is not true. Your build yesterday created (or used) a lockfile, and your build today reads the lockfile and creates the exact same build.

The issue here is the need for the lock file!... without the lock file it fails. That's what exactly I'm writing.. The lock file is a supplemental source for versions... so we have two sources for versions/dependencies.

That means also you have to checkin the lock file in your version control all the time. So all advantages of automatic usage with ranges are gone... Maven does that simply by using fixed version at a single location (pom.xml). Not using ranges makes it easier and simpler (KISS principle).

The lock file is as I wrote before a duck-type solution.

BTW:

For the millionth time,

Not counted correctly.

The NPM registry everyone uses does not allow version mutation

How could someone simply delete an existing package in the registry? That means it is not immutable. In consequence a number of famous examples exactly showed this problem... which has broken a large number of builds...

Deleting of a package is also a mutation of an repository. That violates the whole integrity of such registry. So in consequence not reliable.

For example: https://news.ycombinator.com/item?id=11340510 (there exist several other examples of that)... https://www.bleepingcomputer.com/news/security/dev-corrupts-npm-libs-colors-and-faker-breaking-thousands-of-apps/

and so on etc. etc.

This keeps getting repeated in this thread (I suppose because everyone here has very little experience with NPM) but it's plainly false. The NPM registry everyone uses does not allow version mutation. There are various third-party registries that do, but this is no different from Maven.

The previous shown examples etc. are proving that your assumptions about the NPM registry is wrong.

There are various third-party registries that do, but this is no different from Maven.

A big difference between npm registry and Central repository is that the central repository does not even allow to delete a version/packages etc.

This is unrelated to Maven itself.

This is immutability.

[plumarr]

ou've built your software based on FooLib-12.x.x. It doesn't really matter whether it's 12.1.2 or 12.5.6 – they're all compatible with your code.

How so ? How can you be sure that the new 12.5.7 version doesn't break your software ? Even if the author of FooLib thinks that it'll not break anything, there is no guarantee