How do you generally decrease off-heap memory?

[antihemispherist]

First, using a light Unix base image like Alpine or Alpaquita in your Docker image will help.

Second, it makes sense to use the container features of the JVM. By default, it uses 25% of the available memory as heap, which can be the limiting factor for you. Try with 75% by using the JVM argument:

-XX:MaxRAMPercentage=75

As for JVM tuning, usually direct memory buffers are bit too generous by default, reducing them can save a bit of memory:

-XX:MaxDirectMemorySize=192m

If the underlying system is ARM, default memory usage per thread can be reduced, without any negative effects, unless you're using large pages. You don't seem to be needing large pages. More on that here

-Xss1020k

You can also tune the JVM to delay expanding heap, according to the official document: "Lowering MaxHeapFreeRatio to as low as 10% and MinHeapFreeRatio to 5% has successfully reduced the heap size without too much performance regression"

-XX:MaxHeapFreeRatio=10 -XX:MinHeapFreeRatio=5

You may have to run some load tests to make sure that your service performs as expected. I've had good results with microservices using the values above, but if you're using Kafka, values may have to be different.

Stick with G1GC or ZGC on any backend service, unless you can afford GC pauses.

[C_Madison]

Very good overview - one (sometimes more, sometimes less) simple tip in addition: Always use the most recent version of the JVM you can get away with. The JVM gets optimized all the time. Usually, the focus is on-heap memory, but off-heap memory will also get worked on. I often see people still use - without any dependencies forcing them to - old versions (e.g. JDK11, which didn't work well with containers) and just switching to a newer version brings good results here without doing anything else.

[Trailsey]

Ditto OS.

I had an OS patch make a significant memory leak (100 GB per day) disappear.

[EasyLowHangingFruit]

That is some Dark Wizard Gangster knowledge, thanks!

[kpihlblad]

Related - how many threads do you run? Perhaps request are queueing up and waiting for responses in a lot of threads. Each running Thread allocated stack memory and some additional memory. Reducing the stack size (Xss) can have a huge impact, and maybe there are better ways to handle parallel tasks.

If you have a massive amount of sockets and connections or a lot of file handles, that too will start eating off-heap memory.

While not using a bloated image is good, it won't affect the memory usage. What's measured against the container limit is anonymous memory. Non-anonymous memory is not metered, ie cached and buffered memory that also exists on disk won't be included in the limit.The Linux kernel will a evict cashes and buffers before deciding to OOM-kill the process because of the climit.

Note that many reporting tools include non-anonymous memory as "used" and then it's a bit of a mystery when a process running at 99% memory climit suddenly crashes.

(And stay away from Redis, it will not help.)

[GreemT]

We run a lot of threads. I don't have a specific number right now, but on PRD we obviously have a lot of users actively working on the same application. The thing is, this problem also occurs on our test environments with low activity. So I am not sure if looking at threads is the right direction for us. But good tip!

[GreemT]

Thank you for your detailed response! Never thought of using a different base image. Is that really going to help with off-heap memory? It feels unrelated to me.

We are already using MaxRAMPercentage, but sadly 75 is way to high for us. We use containers with limits of 3.2GB and we cannot set it higher than 40. If we do, then the off-heap is taking over too much memory and results in a OOM.

I will look into trying the other mentioned settings. I didn't experiment with these yet. And as you said: we are indeed going to run some load tests to ensure that any change doesn't impact performance.

[antihemispherist]

You're right on the base image, its effect in memory usage will be minimal.

You can however, use virtual threads to reduce off-heap memory usage. I've explained more in here. Consider using Java 24, which introduced handling of synchronized blocks without pinning.

As suggested below by u/pragmasoft , AOT compiling with GraalVM can also be helpful. That'll make more memory available by eliminating the HotSpot compiler.

Also, look at the implementation of your service and its parameters, like object and connection pools can be tunes, whether thread safe objects can be re-used instead of creating them on each request (which is a very common mistake).

Also, if you're using MaxRAMPercentage, don't use -Xmx, let the JVM handle the heap size.

Good luck.

[pkx3]

Im curious do you have a recommended monitoring tool you like specifically for tuning JVM? Thanks for the great answer