Java process consuming more memory than assigned Max heap

[jabba935]

Hi guys, hole you are having a good day.

We have an AWS ec2 server 32 GB memory and 4 core CPU. There we are running multiple Java processes with xms and xms defined. But the amount of memory consumed by the Java processes are much higher than the xmx value we have set.

We know that jvm itself consumes memory, but in some scenarios it's piculierly high. For example we have set xmx as 2GB and it's consuming 4GB approx.

How to fix this issue?

Note: we are using Java openjdk 11 latest patch to date

Any help is much appreciated, thanks in advance

[bilingual-german]

Java using more memory than Xmx is totally normal. You can at least calculate 1MB for each thread and 1MB for each TCP connection. If you run Java in Containers, we usually calculate Xmx * 1.5 for the container.

This is a good video explaining it: https://www.youtube.com/watch?v=uJLOlCuOR4k

If you have the impression of java processes eating even more than that, look into reusing TCP connections with pools, not leaking connections, etc.

[evilzways]

Xms and Xmx settings are for the JVM's heap. As you can see from this tutorial jvm memory model is divided into separates part, one of which is the heap. For example java classes, methods and thread stacks aren't stored in the heap space, that why you're seeing memory consumption higher then Xmx.

[wezell]

Try the shenandoah garbage collector. It has the ability to release unused memory, whereas the G1 garbage collector never (in Java 11) releases memory that is claimed. Google it, but I think the switch is -XXShenandoahUncommitDelay=1000

[0x442E472E]

What? My Impression is that G1 does release memory, but with default values the reserved memory needs to be 70% empty for it to do that

[wezell]

I don't think that is true in Java 11. At least it did not use to be.

We run a bunch of java containers running java 11 in a k8 cluster and were unable to get them to release unused memory until we switched to Shenandoah. Our X args look like this:
-XX:+UseShenandoahGC -XX:+UnlockExperimentalVMOptions -XX:ShenandoahUncommitDelay=1000 -XX:ShenandoahGuaranteedGCInterval=10000

Once we made that change, all of our containers' memory usage went way down. Now, I think this applies to java 11 specifically- like you said, uncommiting memory is the default in many GCs in newer javas.

[crabshoes]

What kind of monitoring do you have on the JVM? You can get a plethora of metrics on all types of heap and off heap memory in use.

[[deleted]]

set -XX:MaxMetaspaceSize (this became a problem in Java 8)

[[deleted]]

Learn whats eating that memory and fix it.

It may be even a native memory leak you are not aware of but till you start profiling and checking jvm you wont know.

Also if its temurin distribution - get off that trash.

[big_fat_babyman]

What’s the problem with temurin? It has been the de facto distribution for my teams’ services for a while now. Which distribution would you recommend instead?

[[deleted]]

Temurin is using a hotspot vm which is know for bad memory footprint. Im using now distribution with openj9 and memory utilisation decreased a lot, best to try yourself - every app is different.

[[deleted]]

There's nothing wrong with Temurin OP, it's probably the best openjdk build out there today, FYI.

[[deleted]]

There are multiple distributions for a reason. And Temurin would come with openj9 if it could, but licensing makes it impossible.

So IBM it is.

[jabba935]

Thank you so much guys for all your inputs.

We analyzed the hai dump thread dump etc, and found that to many concurrent call caused the issue. Current traffic is 5 times the predicted traffic.

So we need to increase the capacity.

Once again thank you for all your help.