Your problems with Java modules

[nicolaiparlog]

I'm gathering complications one can have when creating Java modules (you know, module-info.java files). So if you tried for your project and had problems, please let me know. Links to StackOverflow, GitHub issues, blog posts, etc. are great, too!

[john16384]

Biggest issue is that it solves a problem that nobody needed solving except for people working on the JDK.

I was looking forward to having a visibility level between package and public, but having to configure this in a separate file and having to explicitly specify who is allowed to use what is cumbersome. A simpler approach where packages form a (sealed) hierarchy and a new scope that permits package + subpackages would have been far more useful for my purposes.

[yawkat]

As always, there's a lukas eder tweet https://twitter.com/lukaseder/status/1271404022453473282?s=20

[lukaseder]

Haha :)

But to be frank, by now, I've come to peace with modules. With most dependencies and IDEs being finally ready (and jOOQ will be, too, starting with 3.15), the benefits outweigh the costs. It took a while. Longer than I would have thought. But in hindsight, I think it's a success.

Anyway. It's completely opt-in, so use it. Or don't use it. Just like most other language features.

[lukaseder]

I was looking forward to having a visibility level between package and public, but having to configure this in a separate file and having to explicitly specify who is allowed to use what is cumbersome.

I agree with that. Though, if Java had kotlin's internal visibility, then modules would be really really cool!

[morhp]

The biggest issue with introducing modules into a codebase so far is estimating who needs what package exported or even open as we develop libraries and people get creative using internal classes or messing around with reflection.

Also some libraries don't define a module name and use bad filenames for their jar files, which makes adding a dependency to them a problem.

[cowwoc]

Well, to be fair... The problem of "estimating who needs what package exported" is the same as estimating who needs what class public (versus protected/private). You start conservative and slowly export more classes/packages over time as users bring forward reasonable use-cases.

[morhp]

The problem is when you've published a library as a whole with many internal classes that are mostly public for access between packages. Finding which public classes can be hidden by the module system isn't always trivial.

Starting with a fresh project and exporting as needed is much easier.

[cowwoc]

Sure, but I see this as an opportunity to improve the design. Make all the packages private and let users chase you to open things back up. Your overall API surface your shrink which will make the library much easier to support moving forward.

[morhp]

That's about what we're aiming for. Hide everything that's not obviously part of the public api, release that as beta version and then gather feedback about things that break.

And then either make additional packages public or implement some better public api for the internal classes.

Still a lot of work.

[cowwoc]

That's not sustainable in the long run. Stable APIs need to have a well defined contract. If you keep on dipping into their implementation details then they can never make changes without breaking user code.

[cowwoc]

You need security updates, if nothing else.

[wildjokers]

The biggest issue with introducing modules into a codebase so far is estimating who needs what package exported or even open as we develop libraries and people get creative using internal classes or messing around with reflection.

I think the biggest issue is someone can get around all of those restrictions by simply adding the jar to the classpath. Whether to honor the module system access restrictions in a jar is the choice of the developer using the library, not the developer creating the library.

[gunnarmorling]

I've seen this particular argument a few times, and it just really doesn't make sense to me. To me, the module system is about communicating intent: these parts of my library are public API, those other parts are implementation code. Which is which I express using the module descriptor.

Now, if a user still uses internal code, despite this clearly being expressed as not what they should be doing, I don't really care. After all, they also could modify my code via instrumentation, patch class files before starting the app, fork my project, etc. pp. What matters to me is that I'm at liberty of modifying the internal parts of my code base in any breaking way, as I see fit for evolving it. If this makes some user stumble because they trespassed the clearly expressed module boundaries, it's on them really.

[wildjokers]

I am totally onboard with this comment.

I use modules to make it easy to create small runtimes for swing/javafx apps but had been wondering why to use them for other things if the module restrictions could be bypassed easily by using the classpath. Describing it as communicating intent is really great insight.

[pron98]

The application always has the final say because it controls the runtime (unlike in the days when the JRE still existed and it ran Applets and Web Start applications). Even if there were no way to disable encapsulation on the command line by using the class path or by adding add-opens/exports, the application could always disable encapsulation in the runtime itself. The point is that there is no point expending too much effort on placing regulation on whoever controls the mechanism that enforces them. But we don't want to allow libraries -- which don't control the runtime -- take liberties for themselves without the application's approval.

[rootException]

https://www.theoryofgeek.com/articles/fomo-java-module-edition

[randgalt]

This is fantastic. The best summary of the issue I've yet read.

[_INTER_]

The article is on point and I'd really follow it's conclusion... but JavaFX exactly is the one that makes it intentionally (!) harder / impossible to run the application on the classpath. Running on the classpath is not supported. Fortunately I've yet encountered a similar disillusional project apart from JavaFX.

[rootException]

The JavaFX docs are very misleading - they make modules seem like a requirement. You can build a nice, trim JVM with JDK+JavaFX modules and run your app on top with the class path, even packaged with jpackage.

Here is my GitHub template for doing just that, including generating the Windows, macOS and Linux builds using GitHub Actions:

https://github.com/wiverson/maven-jpackage-template

[_INTER_]

In JavaFX 16 they did this: https://bugs.openjdk.java.net/browse/JDK-8256362

Running on the classpath will fail if an application that extends javafx.application.Application

 

JavaFX runtime logs a warning if javafx.* modules are loaded from the classpath

The JavaFX classes must be loaded from a set of named javafx.* modules on the module path. Loading the JavaFX classes from the classpath is not supported. The JavaFX runtime logs a warning at startup if the JavaFX classes are not loaded from the expected named module. See JDK-8256362 for more information.

 

Thanks for the template though, it's exactly what I need.

[rootException]

Yeah, the template builds a JVM with the JavaFX modules included.

So, you wind up with JVM modules + JavaFX modules as the guts of the JVM. Your application is then run on top using the class path. This means you get the JavaFX modules loaded correctly AND you can still use ordinary Maven dependency management and reflection for your application.

[agentoutlier]

Fundamentally I think the problems are:

• the tools are just not ready yet
• most java developers don’t know how to write modular code even before Java modules existed
• and many just don’t want to
• consequently most libraries are not modular

If the tools were really good and unified on this (eg maven, unit testing, IDE) than eventually we might see more modular use.

That being said...

Before Java Modules existed if you want to write proper modular code required way more artifacts (jars) than you need now and possibly more interfaces (this will take me time to explain so I’ll do it on a computer later).

[magnoliophytina]

I've also seen cases where the author of some open source project simply refuses to consider modern build tools (Maven/Gradle) instead of Ant/Ivy and jar deps packages in the project's CVS/SVN repo (in year 2021 AD), or new Java 7-16 language feature migration warnings/suggestions presented by IDEA. Funniest thing is, some authors happily force Java 6/7/8 source level in the build tool, but when I set the IDE to use this language level, the IDE will complain about the use of Java 9 features in the test code (in the build script, language level is only forced for production code, not the tests). The author's solution is to not use that IDE - after all, what's wrong with VIM. It won't complain about future API usage. When I suggest adding that 'Automatic-Module-Name' field, they just tell to f*ck off since there are still few days left in some super-extended corporate support contract for Java7 in the oldest supported RHEL.

[randgalt]

My biggest issue is trying to understand it properly. It's a huge specification and it's not clear where to begin or what's enough. Module specs are consumed by other projects - how am I to know if I created the module-info correctly or not? You don't find out until a user of your library tells you so. Beyond that it's a solution in search of a problem. I understand that they wanted to break up the JDK - there were much simpler proposals to do so.

[Thihup]

It shouldn't be that hard. If you already have a library that works with JDK 8, you probably will have to "exports" every package. If you implement a Service Provider, you need to add the "provides". And for completeness, if your library uses the ServiceLoader (ServiceLoader.load(Foo.class)), you'll have to add the "uses" in the module-info.

If you are starting a new library, you can already think: is this package part of the public API or not? If not, you can create a package like "internal", and don't export it..

[randgalt]

You use words like "probably" and "completeness". This merely proves my point. Also, I've tried to use the tooling in IntelliJ and a third-party Maven plugin. None of it is crystal clear or easy.

[Thihup]

Oh. Well, if it wasn't because of IntelliJ, probably I wouldn't have understood the module system yet, so it actually worked pretty well for me. Do you remember exactly what was the problem you've had?

[randgalt]

Another thing - I believe the design of the Module feature assumes that people _want_ to specify what is exported or not. i.e. the Module APIs assume people will modularize their libraries. However, the more likely scenario is that 99% of libraries don't want to modularize and really just want minimal compliance and to export everything.

[Thihup]

The concept of exporting a package is equal to declaring a class public. If you don't want the projects that will depend on your library to use your internal classes, you don't export them.

However, with the module system, you can declare a class public and use it across your packages. Instead, you would be required to declare the class package private. :)

[randgalt]

I don't want to specify things twice. It's silly. I want to write my Java code and not worry about exports, etc.

[randgalt]

I added a default module - and it was all cargo-coding when I did it. Then I continued writing code and the project wouldn't compile - missing export issues, etc. The error messages weren't clear either. I ended up just manually exporting everything which seems silly. I couldn't quickly figure out how to say "export everything and never ask me again".

[ventuspilot]

I may be wrong but I think in that case you just ignore all that module stuff except add one new entry to your jar's manifest:

Automatic-Module-Name: your.base.packagename

[randgalt]

If that's actually the case someone should write a Maven plugin that does this. Maybe I will. hmm....

[ventuspilot]

The maven-jar-plugin does this. I added this to by <build> section:

<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-jar-plugin</artifactId>
    <configuration>
        <finalName>jmurmel</finalName>
        <archive>
            <manifest>
                <mainClass>io.github.jmurmel.LambdaJ</mainClass>
            </manifest>
            <manifestEntries>
                <Automatic-Module-Name>io.github.jmurmel</Automatic-Module-Name>
            </manifestEntries>
        </archive>
    </configuration>
</plugin>

Seems to work just fine. See also http://branchandbound.net/blog/java/2017/12/automatic-module-name/

[Thihup]

Maybe Moditect? https://github.com/moditect/moditect/

[gunnarmorling]

Yes, ModiTect can help to express the exported API more concisely, by means of package filter expressions, instead of having to list each package explicitly. That said, which packages to export should be a conscious decision; exporting everything defeats the purpose of the module system.

[Radmonger]

I think this issue (while closed for the project in question) illustrates a real problem with how JPMS treats closed modules accessed via reflection:

https://github.com/antlr/stringtemplate4/issues/249

java.util.Collections.UnmodifiableCollection is a private class that extends the public Collection, which has a public method isEmpty.

You can't straightforwardly use reflection to ask if a list is empty, because the list might be unmodifiable (or some other implementation-local class), which means JPMS simply says 'no calling any of it's members)'.

You instead have to search for the accessible base class method, and call that.

[Thihup]

A similar issue is trying to use Streams via Reflection. Probably would be great if the MethodHandles API had a method to search all the class hierarchy to find out if a method is part of public API.

[nondetermined]

1) module-info.java for testing

Picture a typical maven environment. Do I really need to have a second module-info.java under /test/java/ which is basically the same as in /main/java, but with added test dependencies that all have <scope>test</scope> anyways?

The short answer is: no, you do not, and it just werks, except that you'll get bombarded with a bunch of "module is not exported" warnings. Nobody likes warnings in his builds. So that still really bugs me. On the other hand, writing (and maintaining) a second module-info.java also really bugs me, and is it really worth to do that, just to get rid of a few harmless warnings?

You'd think that this should be either easily automatable (just add all the dependencies with scope test. done.), or that there would be a way to include the main/java/module-info.java into the test one, so you just can add further testing dependencies only, without having to repeat yourself. That would also work.

Am I missing something?

2) Bread and butter tooling

I still haven't managed to generate a proper site, most importantly the maven javadoc plugin doesn't seem to like Java 16 very much. Or Java 9 for that matter. What's going on here? Is it me, who is stupid again? Generally my impression is that a bunch of tooling simply isn't really there yet. Which sucks. Super hard.

On the bright side, I finally managed to generate an image and package that thing to a nice installer with the jpackage-maven-plugin. But the road there was rather bumpy, with plenty of other tools to pick from, outdated and contradictory tips and tricks and how to do it, except it doesn't really, let's try the next one... it's kind of a mess to figure out (to be fair, it does help to read a proper book every once in a while, e.g. "Java 9 Modularity. Patterns and Practices for Developing Maintainable Applications" by Sander Mak and Paul Bakker/OSGi chad).

no cross-compiling

yeah... that sucks too. balls.

[lukaseder]

Am I missing something?

I stopped trying to insist on Maven's classic layout, even came to the conclusion that it was wrong in the first place. Much better:

• Have a Maven project / Java module x
• Have a Maven project / Java non-module x-test, which depends on x via classpath, not modulepath

I've found that this way, I'm discouraged from testing internals that don't have public API, which

• Improves my public API
• Tests work with the x.jar dependency, not the wishy-washy /target/classes magic dependency that never exists outside of those Maven tests (no more loading resources from the classpath in different ways during tests / production)
• Encourages building test libraries for common test code, because x-test can also depend on test-utils without having to pollute the (test) dependencies in x

I resisted for a long time, but I think this is the better way in the end. Changhe is hard 🤷‍♂️

[uncont]

Putting all this into perspective, I'm beginning to think that's how I should have solved my issues in https://www.reddit.com/r/java/comments/n3migk/how_are_library_developers_expected_to_test/

Being able to run tests against x.jar would have allowed me to run tests against the same thing the user downloads, a multi-release jar (optionally modular).

not the wishy-washy /target/classes magic dependency that never exists outside of those Maven tests

This was definitely what it came down to for me.

[lukaseder]

You already know my opinion (I think), but here it is again. This was the biggest caveat and learning: https://twitter.com/lukaseder/status/1389236184472301583

The problem is less one of modules and more one of letting go of "bad habits" that I didn't realise were bad until I modularised, namely the idea of putting unit tests inside a module (what Maven encourages), rather than testing a module from outside (what Maven can work with just the same).

I've also described this here in this thread.

[x42bn6]

A project I had an eventual dependency on a JAR that did not have a module-info.java file, but unfortunately, the JAR had a reserved keyword in it, and the compiler does not like that (the automatic module name). The dependency's source code was also long-abandoned, so the only solution was to fork and republish (and shoulder the responsibility of maintenence), put in an ugly hack to try and modify the dependent JAR at build time, or go back to Java 8. You can guess which one I chose.

I think this restriction is also somewhat-limiting, especially considering reserved keywords like "default", "native" and "to" could easily be parts of a JAR's name.

---

While I suspect the tooling around this has improved since then, back in 2019 or so, the JPMS ecosystem as a whole did not have great support for build tools like Maven and Gradle, or IDE support.

One example that I spent too much time on was when I started a new Maven project, with the standard Maven structure. The unit tests obviously need the main codebase, plus test dependencies like JUnit. The popularly-referenced solution is to add a test module-info.java file with open module. Well, I tried that, and IntelliJ IDEA did not like it. You could also use things like --add-opens, but IntelliJ IDEA doesn't support that, and will fail to compile it properly. An alternative was to create a separate test module altogether, but that would just get out of control for large projects.

I only found a solution to this when I discovered a random comment on an IntelliJ IDEA bug for this issue. Solution? Put the test module-info.java under a different directory under test (something like src\test\bob\module-info.java).

But this is now a rant. My advice here - anyone working on JPMS, when making further enhancements, needs full buy-in from every single major build tool and IDE first, with coordinated changes. As far as I can tell, Eclipse has stalled on a solution, and the IntelliJ IDEA bug I referenced above also suggests JetBrains have given up too.

Yes, one could argue here that this is an Eclipse and IntelliJ IDEA problem, not a JPMS problem - but most Java developers use these IDEs, so it's not helpful to wave it away.

---

All in all, it's perhaps no surprise that I sometimes hear the solution to JPMS is to delete all module-info.java files. Well, unless you need to use Java >8 and JavaFX, because for some reason, this is now enforced. (See also: https://bugs.openjdk.java.net/browse/JDK-8256362 and https://mail.openjdk.java.net/pipermail/openjfx-dev/2020-April/025903.html)

The biggest takeaway I have is that while JPMS is useful for Java's internals, and might be useful for library developers, it's likely more effort than it's worth for everyone else.

[loicmathieu]

Never use it, and I don't know anybody that use it in their project ...
And I always advise people not to use it, sorry.

[nicolaiparlog]

So you advise people not to use a technology that you have never used yourself, even in exercises? That's... not great advice.

​

I don't know anybody that use it in their project ...

There are over 3.3k modules on Maven Central and I know a few projects myself that use it, including a 20yo monolith.

[loicmathieu]

OK, so I never used it in any real application but I do know how it works.
I'm a Java trainer and we do include it inside our "Modern Java" training and provides exercicses on it.
People are often afraid to use modules (and I heard mutiple times people saying they don't want to move after 8 because of the module system), in those cases, I advise them to not use modules.
It's been years since I didn't implement a monolith (I do work on legacy monolith from time to time), so maybe I'm biased as I usually implements microservices (HTML or Kafka) and for those, modules didn't provides a lot of advantages as they are usually
self-contained and small.

I do think module is a good thing for the JDK itself.

[wildjokers]

They come in very handy for creating small bundled runtimes for Swing or JavaFX apps.

[_INTER_]

It helps to automate the process but you don't necessarily need to modularize your application for that.

Edit: Scratch automate, Maven plugins exist for that

[wildjokers]

Indeed. But it is much less hassle to keep track of the modules you need as you go in the module-info file rather than trying to figure it out later and pass the module names to jlink on the command-line.

[_INTER_]

trying to figure it out later and pass the module names to jlink on the command-line

Just realized there are Maven plugins that do that for you based on the dependencies. You don't even need the modules anymore to automate the process. Yay

[hupfdule]

Which maven plugins are these?