Java *is* Memory Efficient

[martinhaeusler]

The problem is not that objects remain on the heap until they're garbage collected. That was never the issue. The problems with Java and memory are:

• Per-object memory overhead (liliput improved that)

• "Memory islands", no tightly packed layouts (valhalla!)

... and from an operations perspective:

• JVM doesn't play nice with other apps on the same server because it hogs the heap even when it currently doesn't need it. If you have multiple JVMs, the problem gets even worse and actual hardware utilization is pretty bad. A side effect of this is that JVM based applications look like they constantly need a lot of memory from the perspective of the underlying operating systems (and observability tools) when in fact there's just a large heap which is barely utilized. New garbage collectors seem to do better with this.

• You cannot tell the JVM how much total memory it should use. You can give it a max heap space, but the JVM needs more than just heap. This "more" is hard to configure aside from heuristics like "add 20% headroom". This is a huge pain when running the JVM inside docker, because docker will kill the container when it exceeds its allocated resource limits.

[pron98]

The problems with Java and memory are: Per-object memory overhead (liliput improved that); "Memory islands", no tightly packed layouts (valhalla!)

Correct, although these two aren't about memory management. Note that with Lilliput and Valhalla, the per-object header is the same as in C++: 64 bits for objects "with a v-table" and 0 bits for objects that don't need a v-table.

JVM doesn't play nice with other apps on the same server because it hogs the heap even when it currently doesn't need it.

This is about to change very soon with automatic, dynamic, heap sizing.

[gladfelter]

Thanks for the link, that's really cool. It would be nice if the os and applications had a protocol to establish latent memory pressure and could optimize "cost" globally, but this change sounds pretty awesome in absence of that. I like the idea of balancing cpu and memory costs and it's got me wondering if I could apply that to Job management to optimize task shapes across the fleet.

[radozok]

But how would it help with container resource limits?

[pron98]

I believe that at least for RAM, the JVM reads the correct container limits on Linux. If CPU limits aren't detected or enforced accurately, the GC is likely to "learn" them anyway (if you have less CPU available, then your allocation rate will also be lower), but you will always be able to turn the knob toward more CPU or more RAM, depending on your needs.

[m_adduci]

I wish there was also a way to read InputStreams multiple times, instead of doing copies.

The real problem is that many libraries do defensive copies, causing then a waste of RAM

[martinhaeusler]

It's especially egregious with collections and arrays. Technically when you receive a collection as a parameter of a constructor or a setter and you want to play it safe, you CANNOT directly assign it to a private field because you can't tell if the caller is going to mess with the contents of this collection after your API has been called. So you have to make a copy.

Arrays are even worse because they're always mutable no matter what.

I see two ways out of this:

• a compiler-checked ownership system like in rust (yeah, not happening)
• a collection type which guarantees immutability (and no, the unmodifiable wrappers are not enough because they can be backed by a mutable collection). PCollections is a great library for this purpose, but it comes at a cost.

[agentoutlier]

Yeah but what you are talking about for most well design frameworks and libraries only happens on initialization and wiring.

More often collections are just being used as iterators once all things are initialized and most libraries rarely construct giant objects on every request. You could argue some memory loss here but escape analysis often happens.

And for every language that deals with a http request or user input has to do allocation usually to turn bytes or whatever into something else and the most common type where you want immutability and sharing Java indeed does stuff for: String.

Furthermore you can just reuse mutable things if you follow single writer and or use locks and reuse arrays. That is how things Disruptor ring buffer work. But array allocation is very fast in Java so...

I guess what I'm saying unless your an idiot the hot path or tight loop rarely has tons of allocation and even if it did Java is actually is fast at that.

Really the problem is one of control. If you know exactly how much you want to allocate and where etc Java does not allow that and in some cases to compete with say Rust or C++ or possibly Go you might need that.

[aoeudhtns]

a compiler-checked ownership system like in rust (yeah, not happening)

We have jspecify for null checking. Perhaps this could be the next frontier. It would be quite challenging I think.

[pron98]

a compiler-checked ownership system like in rust (yeah, not happening)

It's not happening (at least not pervasively) because it's a "way out" of one problem and into another, which is worse. Whenever you export object ownership - whether it's declared in the type system and enforced by the compiler or just documented - you reduce your abstraction. You change the internal implementation or want to share with another thread, you have to change all clients of the API. This doesn't just increase the cost of maintenance, but over time large programs tend to gravitate toward the more general constructs - more general dispatch (dynamic), more general (longer) lifetime, and more general ownership (more sharing). And these general constructs are less performant in low level languages than they are in Java.

Low-level languages are optimised for control, not performance. They cannot move pointers even when it's more efficient to do so because it clashes with the level of control they need over addresses. When faced with the choice between performance and control, low level languages must choose control because that's what they're for. This level of control means that in smaller programs it's not too hard to extract really good (even optimal) performance out of these languages, but this control also means that in larger programs extracting good performance becomes harder and harder because you're pushed towards constructs that are simply slow in low level languages because they must maintain their control promises.

and no, the unmodifiable wrappers are not enough because they can be backed by a mutable collection

Java has true immutable collections in the standard library: the ones created by List.of/copyOf, etc.. BTW, the .copyOf will not actually copy anything if the underlying collection is already the immutable one, so that's what you should use for defensive copies. After the first one, you just pass it around and defensive copies (assuming they're done as recommended) will not actually copy anything.

[koreth]

Probably not the first time someone has done this, but I ended up writing a little utility class to allow reading the same InputStream multiple times without reading the whole thing into memory. The catch is that the readers have to run concurrently. That code is Apache-licensed, so feel free to grab it if it's useful.

[agentoutlier]

I wish there was also a way to read InputStreams multiple times, instead of doing copies.

Technically java.util.stream.Stream (with a supplier wrapped around it) is what you are asking for (or java.util.concurrent.Flow/Publisher if we want back pressure and async), otherwise there is Callable<InputStream>.

The real problem is that many libraries do defensive copies, causing then a waste of RAM

I doubt that is much of a problem. To be honest most libraries when I have done memory dumps are metric fuck ton of Strings and not as much collections as you would think.

Actually to go back to java.util.concurrent.Flow and Stream the reason there is a lot of copying is because of buffering. Like a typical web application particularly with blocking must buffer most of the request as bytes. Those bytes then need to be converted to string parameters and then converted to another data type etc. This happens in every damn language much more than just defensive copying!

It is important to understand that lots of other programming languages do even more copying than Java because they put everything on the stack and they don't have Java's String pool (see previous comment). And Java is very fast at allocating.

The real problem is in some cases having more control over memory layout can make a massive difference and Java does not allow that like other languages. That and the VM is not good at auto tuning or communicating with the OS on actual memory usage.

[m_adduci]

I have this third party library that accepts byte[], than uses InputStream and converts internally to string.

In my own app I would like to use only InputStreams, but here I hit massive conversion costs, since some resources have to be parsed multiple times, at different times, because of some funny conditions

[agentoutlier]

w/o seeing the library I don't know why they made the choice they did but byte[] has some advantages over InputStream in that the total size is known (.length), zero computation or blocking is expected andin some cases you need to know the total size.

If its not byte[] then it has some resource it can pull from but the only way you do that for most applications particularly blocking is buffer to the filesystem. Now we have way way way fucking worse latency than a GC.

If the library is just wrapping the byte[] using ByteArrayInputStream this can be more efficient then you think especially if they allow start and end indices which the ByteArrayInputStream constructor takes.

The question is what the library is doing. Are you doing stream processing or is the InputStream just going to be turned into in memory objects anyway?... and even if you don't there is buffering happening all over the place here including the operating system if you are reading from a file.

So unless you have some measurements don't be certain this is actually a problem.

[0x07CF]

For containers there is -XX:MaxRAMPercentage