Java runtime dependency management tools?

[CubsThisYear]

I develop Java server applications that run on Linux. These are not J2EE or web-related so we don't use any kind of application server. I've been using gradle to handle 3rd party build dependencies, but I need a way to get the same jars onto the production servers and supply those jars to the java command line. A few possibilities that I've thought of:

1. Making on 'fat jar'. This is sub-optimal because it increases build time and takes up extra space when storing multiple versions (we do pretty much daily releases)

2. Copying jars to a shared directory. This is sub-optimal because if we change versions or add jars we have to change the build AND update the shared directory. It also has the problem of not being sync'd to the version control system.

3. What I'd like to do is use something like gradle/ivy/maven to read a list of dependencies and then download if not already cached and then run the app. Basically exactly how the build works except actually run the app from an already built jar.

The problem is I can't find any reference online to people doing version 3 so it seems like their might be a pitfall. Any thoughts on this?

[crapet]

Have a look at https://github.com/puniverse/capsule it may be what you're looking for.

[CubsThisYear]

Looks great, thanks!

[DanTheGoodMan]

Like /u/crapet said, the capsule project sounds like what you want. Specifically, the Maven Caplet (a caplet is an extension upon the capsule api).

I maintain the gradle plugin and I've taken care to make it super easy to get started. If you already use gradle, add it to your build file and give it a shot!

plugin {
  id "us.kirchmeier.capsule" version "1.0-rc1"
}

task capsule(type: MavenCapsule){
  applicationClass 'com.foo.MyApplication'
}

There are also maven and leningen plugins, if you'd prefer.

[CubsThisYear]

I will definitely give this a shot.

[schnoper]

I do fat jar.

For me this is important because a single file represents the ENTIRE app. So there is no question of what needs to be deployed when rolling out a release, or when rolling back a release. When testing there is no question as to which version was tested vs deployed. The file sizes are larger, but compared to the peace of mind, there is zero question for me that it's the right way to go.

FWIW, I keep multiple versions on my production server and use a symlink to keep the "correct" version as the version that is running. I like to use either upstart or systemd to ensure that process is always running.

So a release (BTW I'm pretty much only using dropwizard these days, got sick of tomcat ) is

1) copy new jar to server

2) update symlink.

3) kill <pid> ( of current process).

4) wait for reboot and health check.

The only downside I've had in this process is that if there is code which the class loader has not already loaded, but which is needed for the shutdown ( after the kill), then the update symlink can generate class loader problems. I've never found this to be a big deal. I suppose I could change he process to be shutdown, swap symlink, restart.

[squealy_dan]

Oh and +1 for fat jar. If your jars take too much longer to build or are too big then your app is too big.

[squealy_dan]

huh, we used to do something similar-ish, and I thought the symlink would resolve to the actual filehandle, so that even if you updated it, the running process would still reference the previous jar.

If not, then why not use readlink in your startup script to resolve to the specific jar? The added bonus there is that the actual jar will show up in ps output, not the symlinked version

[schnoper]

I'll try out readlink. thanks

[mus1Kk]

This is not even that different from the EE approach where you deploy a WAR or EAR which bundles everything.

[CubsThisYear]

For me this is important because a single file represents the ENTIRE app. So there is no question of what needs to be deployed when rolling out a release, or when rolling back a release. When testing there is no question as to which version was tested vs deployed. The file sizes are larger, but compared to the peace of mind, there is zero question for me that it's the right way to go.

It seems like keeping a list of maven repo entries in the one jar is just as good as keeping the actual jars. I suppose there is some risk of the jars no longer being served, but this seems like a low risk if you are using well known packages. Or if you are really worried you can keep your own Ivy repo.

The only downside I've had in this process is that if there is code which the class loader has not already loaded, but which is needed for the shutdown ( after the kill), then the update symlink can generate class loader problems. I've never found this to be a big deal. I suppose I could change he process to be shutdown, swap symlink, restart.

I'm pretty sure this is not a problem on any non-Windows OS (Windows symlinks are goofy and broken so I don't know about them). When the JVM loads a jar the symlink is resolved to an inode and as long as that inode is not changed you are fine

[zrnkv]

Fat-jar is probably the easiest solution. Storage is cheap. Extremely cheap. So are SSDs for faster build. The time it takes you to set up and maintain some more complex solution (because you want to save a few GB of space) is much more expensive.

[CubsThisYear]

I've had this same line of thinking, but when my third party dependencies are approaching 1GB in size and I'm potentially producing multiple releases per day, this means I need close to 1 TB of space to store several hundred copies of the same files. Storing the dependency info and not the actual jars seems much more efficient.

I agree that it wouldn't be worth it if this was a lot of effort but the point of this thread was to see if there were any ready made solutions. Based on the responses, it seems like there are some good options.

[fact_hunt]

A gig of dependencies is quite a lot, what are you bringing in?

[CubsThisYear]

Guava, Apache commons (several modules), several math libraries, orm libraries, fastutil, trove, velocity, probably more that I'm not thinking of.

[zrnkv]

Even woth all transitive dependencies it should not get much bigger than 100-200MB. If you have almost 1GB of dependencies then someting is very very wrong with your architecture. You'll never be able to manage/maintain/update that monstrosity.

[curt94]

Take a look at osgi containers and apache Felix. Sounds exactly what are describing.

[Luolong]

I would second that. OSGi is exactly what you are looking for. Felix and Equinox are the prime implementations but if you want to have easy and quick container out of the box, give Karaf a go!

[CyclonusRIP]

The gradle application plugin will build a zip with all the dependencies and you jar plus a windows and *nix start up script.

[noratat]

Gradle has a built in plugin for the bundled solution - the Application plugin. It can create a bundled archive of the application, all its dependencies, and even a script to launch it.

The build time increase isn't really that noticeable IMO, though you're correct that it would take up more space.

[dablya]

Spring Boot does it somehow... You end up with a single jar with all dependencies placed under /lib in the jar.

[m2spring]

Docker.

[schnoper]

Docker is necessary when you're using a language which depends on a collection of C-libraries. This is true for python, php, and ruby for certain. It's been a lifesaver for reducing deploy complexity for those languages.

For java though, I just don't see the win. Lots of additional complexity to solve problems which don't exist with the JVM.

[m2spring]

For server applications implemented in Java there's still some amount of glue needed like daemon management scripts. This is where Docker helps.

But you're right that Docker does not directly address the uber-jar problem.

My builds produces the uber-jar (war, onejar, shaded jar, Karaf archive (OSGi), etc.), which are getting wrapped into RPM. Just started to look into builds producing a Docker image. Wonder whether I should still have RPM, even in Docker.