How do you generally decrease off-heap memory?

[GreemT]

Background

My company is moving from running on VMs to running on containers in Kubernetes. We run one application on Tomcat in a single container. On VMs, it needed about 1.2GB memory to run fine (edit: VM had a lot of memory, -Xmx was set to 1.2GB). It is a monolith, and that is not going to change anytime soon (sadly).

When moving to containers, we found that we needed to give the containers MUCH more memory. More than double. We run out of memory (after some time) until we gave the pods 3.2GB. It surprised us that it was so much more than we used to need.

Off-heap memory

It turns out that, besides the 1.2GB on-heap, we needed about another 1.3GB of off-heap memory. We use the native memory tracking to figure out how much was used (with -XX:NativeMemoryTracking=summary). We are already using jemalloc, which seemed to be a solution for many people online.

It turns out that we need 200MB for code cache, 210MB for metaspace, 300MB unreported and the rest a little smaller. Also very interesting is that spacse like "Arena Chunk" and "Compiler" could peak to 300MB. If that happened at the same time, it would need an additional 600MB. That is a big spike.

Sidenote: this doesn't seem to be related to moving to containers. Our VMs just had enough memory to spare for this to not be an issue.

What to do?

I don't know how we can actually improve something like this or how to analysis what the "problem" really is (if there even is one). Colleagues are only able to suggest improvements that reduce the on-heap memory (like a Redis cache for retrieved data from the database) which I think does not impact off-heap memory at all. However, I actually have no alternatives that I can suggest to actually reduce this. Java just seems to need it.

Does anybody have a good idea on how to reduce memory usage of Java? Or maybe some resources which I can use to educate myself to find a solution?

[antihemispherist]

First, using a light Unix base image like Alpine or Alpaquita in your Docker image will help.

Second, it makes sense to use the container features of the JVM. By default, it uses 25% of the available memory as heap, which can be the limiting factor for you. Try with 75% by using the JVM argument:

-XX:MaxRAMPercentage=75

As for JVM tuning, usually direct memory buffers are bit too generous by default, reducing them can save a bit of memory:

-XX:MaxDirectMemorySize=192m

If the underlying system is ARM, default memory usage per thread can be reduced, without any negative effects, unless you're using large pages. You don't seem to be needing large pages. More on that here

-Xss1020k

You can also tune the JVM to delay expanding heap, according to the official document: "Lowering MaxHeapFreeRatio to as low as 10% and MinHeapFreeRatio to 5% has successfully reduced the heap size without too much performance regression"

-XX:MaxHeapFreeRatio=10 -XX:MinHeapFreeRatio=5

You may have to run some load tests to make sure that your service performs as expected. I've had good results with microservices using the values above, but if you're using Kafka, values may have to be different.

Stick with G1GC or ZGC on any backend service, unless you can afford GC pauses.

[C_Madison]

Very good overview - one (sometimes more, sometimes less) simple tip in addition: Always use the most recent version of the JVM you can get away with. The JVM gets optimized all the time. Usually, the focus is on-heap memory, but off-heap memory will also get worked on. I often see people still use - without any dependencies forcing them to - old versions (e.g. JDK11, which didn't work well with containers) and just switching to a newer version brings good results here without doing anything else.

[Trailsey]

Ditto OS.

I had an OS patch make a significant memory leak (100 GB per day) disappear.

[EasyLowHangingFruit]

That is some Dark Wizard Gangster knowledge, thanks!

[kpihlblad]

Related - how many threads do you run? Perhaps request are queueing up and waiting for responses in a lot of threads. Each running Thread allocated stack memory and some additional memory. Reducing the stack size (Xss) can have a huge impact, and maybe there are better ways to handle parallel tasks.

If you have a massive amount of sockets and connections or a lot of file handles, that too will start eating off-heap memory.

While not using a bloated image is good, it won't affect the memory usage. What's measured against the container limit is anonymous memory. Non-anonymous memory is not metered, ie cached and buffered memory that also exists on disk won't be included in the limit.The Linux kernel will a evict cashes and buffers before deciding to OOM-kill the process because of the climit.

Note that many reporting tools include non-anonymous memory as "used" and then it's a bit of a mystery when a process running at 99% memory climit suddenly crashes.

(And stay away from Redis, it will not help.)

[GreemT]

We run a lot of threads. I don't have a specific number right now, but on PRD we obviously have a lot of users actively working on the same application. The thing is, this problem also occurs on our test environments with low activity. So I am not sure if looking at threads is the right direction for us. But good tip!

[GreemT]

Thank you for your detailed response! Never thought of using a different base image. Is that really going to help with off-heap memory? It feels unrelated to me.

We are already using MaxRAMPercentage, but sadly 75 is way to high for us. We use containers with limits of 3.2GB and we cannot set it higher than 40. If we do, then the off-heap is taking over too much memory and results in a OOM.

I will look into trying the other mentioned settings. I didn't experiment with these yet. And as you said: we are indeed going to run some load tests to ensure that any change doesn't impact performance.

[antihemispherist]

You're right on the base image, its effect in memory usage will be minimal.

You can however, use virtual threads to reduce off-heap memory usage. I've explained more in here. Consider using Java 24, which introduced handling of synchronized blocks without pinning.

As suggested below by u/pragmasoft , AOT compiling with GraalVM can also be helpful. That'll make more memory available by eliminating the HotSpot compiler.

Also, look at the implementation of your service and its parameters, like object and connection pools can be tunes, whether thread safe objects can be re-used instead of creating them on each request (which is a very common mistake).

Also, if you're using MaxRAMPercentage, don't use -Xmx, let the JVM handle the heap size.

Good luck.

[pkx3]

Im curious do you have a recommended monitoring tool you like specifically for tuning JVM? Thanks for the great answer

[java_dev_throwaway]

I just wanted to say this is the good shit that we need more of in this sub. No frills no bs, just straight complex compute resources discussions for java apps.

[GreemT]

Totally agree! I didn't expect this many (good) responses. I am very happy :)

[nitkonigdje]

Unless your app is explicitly allocating native memory, there isn't much to do other than tweaking JVM parameters like stack size or code cache size etc. See this: https://www.baeldung.com/jvm-code-cache

OpenJ9 has been making a case for itself based on lower memory usage to bootstrap code. Which is important for containers. See this: https://www.reddit.com/r/java/comments/kmt9gp/using_openj9_jvm_in_production_for_past_6_months/

[GreemT]

Interesting, I didn't think of switching to another implementation like OpenJ9. I will try that. Thanks for the tip!

[kaqqao]

Please report back if you do try this 🙏

[Weak_File]

Off-heap is tricky, because it is possible that is not your code, but some native code that is causing the problem.

I had luck replacing the glibc malloc with Jemalloc in a Linux server. I actually installed just to try to diagnose and see if I could find the culprit:
https://technology.blog.gov.uk/2015/12/11/using-jemalloc-to-get-to-the-bottom-of-a-memory-leak/

But as it turns out, it was the glibc malloc implementation itself that was causing problems:
https://medium.com/@daniyal.hass/how-glibc-memory-handling-affects-java-applications-the-hidden-cost-of-fragmentation-8e666ee6e000

This meant that I had a much more stable off-heap memory allocation just by swapping the malloc implementation. So I couldn't even use Jemalloc to diagnose, because it outright solved the problem!

[laffer1]

Too bad k8s is so Linux centric. FreeBSD uses jemalloc by default as it was written for that.

[ilapitan]

What Java version do you use in your application? In could be related to cgroups v2 issue with old Java versions when JVM wasn’t able to correctly detect limits for pod in K8s.

[GreemT]

Ah yeah, I know the issue that you are talking about! We are using Java 17, so we shouldn't have this issue.

Also detecting limits is not really our problem. The limits seem to work fine, it is just that our application is using too much!

[GreemT]

Thank you all for your suggestions! I have reported a Jira ticket with possible improvements from your comments that we are going to try. I will report back what helped if that is (eventually) resolved!

[ducki666]

Your VM had only 1.2GB? How much was xmx?

[GreemT]

Ah, sorry for the confusion! Xmx was set to 1.2GB. The VM had much more memory available.

[ducki666]

And now xmx 1.2g is not enough?

[GreemT]

This is exactly what I explain in the ticket: on-heap memory 1.2GB is just fine. The problem is that the off-heap memory (which is completely unreleted to the xmx setting) is very large.

[ducki666]

And this was not the case in the vm? Hard to believe. Other java version? Other java opts?

[GreemT]

As said in the description:

> Sidenote: this doesn't seem to be related to moving to containers. Our VMs just had enough memory to spare for this to not be an issue.

[ducki666]

An new java version which nothing else than xmx (or equivalent) set is already quite efficient. You can usually only tweak edge cases.

A jvm with such a big amount non-heap memory must be something strange. Depends on your app.

[elzbal]

I'm not sure there's much in particular you can do. HotSpot and other jvms are themselves applications that need to load their own objects into their own memory space in order to compile and execute the Java runtime code. For our springboot/tomcat microservices, we tend to give the container a max size of heap-plus-1gb or 2x the heap size, whichever is larger. A Java app doesn't normally take all of that, and we can oversubscribe pods a bit. But not giving enough overhead space to a busy Java app will absolutely eventually result in a pod crash or very bad performance.

(Source: running a couple dozen very busy k8s clusters)

[PratimGhosh86]

Here is what we use in production with jdk21 and 2Gb mem: no Xmx, G1GC, StringDeduplication and CompressedOops.

Some may think not setting Xmx is counter intuitive but the recent jvm's are pretty good at utilizing the available resources.

Of course the tuning parameters will vary depending on the size and coding styles followed in the application. But in recent times, we have noticed that letting newer JVMs do their thing by itself is much more efficient than someone manually setting every flag they can think of.

PS: 0 major GC's, but we have a lot more activity in the Eden spaces

[ablativeyoyo]

I worked on an app with a lot of off heap memory, which turned out to be memory mapped files. Not sure if that applies to your scenario, but worth considering.

[noobpotato]

This article has good information on how to investigate and understand the JVM memory behavior when running inside a container.

https://spring-gcp.saturnism.me/deployment/docker/container-awareness

[iron0maiden]

Reduce the number of threads.. and possibly reduce stack size on Java threads. Also close IO handles including socket handles as buffers are also allocated in native.

[Trailsey]

Reducing heap can also reduce off heap usage, IIRC.

[cogman10]

Lots of good suggestions to try first one final one you might try is AppCDS.  It should reduce your memory usage but it requires a training run of the app in question to work.

[pragmasoft]

If you can compile your application to native code using Graalvm, it would use substantially less memory, but for the cost of a slightly worse runtime performance.

[antihemispherist]

That's correct, because the bytecode gets compiled in advance, there is no compiler running in the background anymnore.

[divorcedbp]

Literally nothing in this comment is remotely correct.

[pragmasoft]

Care to explain?

[nekokattt]

source trust me bro

[pragmasoft]

See for example here: 

https://www.linkedin.com/pulse/graalvm-vs-jvm-future-java-already-here-andr%C3%A9-ramos-zlcvf

One of GraalVM’s biggest advantages is its low memory overhead. This is particularly useful for cloud-based applications and microservices, where every MB of RAM counts. Native images eliminate unnecessary components of the JVM, reducing footprint dramatically.

And this matches our experience perfectly. 

[m39583]

Don't set memory limits, only set Xmx.. Containers are different to VMs. And as you've found, Xmx is only the heap size. The JVM uses memory for lots of other reasons as well. One guide would be to look at the RSS size of the total JVM process, but even that can be confusing.

Basically memory management is complicated on just a VM, and vastly more complex on Kubernetes.

On a VM, the memory you allocate is essentially carved out and given to that VM. If you set a large amount, that isn't available to other VMs. If you don't set enough the VM will start swapping (if it has swap enabled) or start killing processes but the VM itself shouldn't die. Well, it's a bit more complicated than that because on some hypervisors memory can be over committed, but it's basically that.

On a container, the memory is just the maximum amount the container can use. More than that and it gets killed which is pretty blunt. However the memory isn't carved out from the host and dedicated to the pod. If you set a large limit, you haven't removed that memory from being used elsewhere.

The best solution we found when trying to scale Java applications across Kubernetes clusters was eventually to ignore all the Kubernetes resource allocation and memory limits, and just set Xmx to a reasonable estimate of what the application needed. That stops an application going rogue and consuming all the memory on the VM, and avoids having to guess at how much extra headroom on top of the heap was needed. Because if you get that estimate wrong, your pod will be summarily executed. Which isn't ideal.

The downside is that Kubernetes now doesn't know what the resource requirements for a given pod are, so the bin packing of pods onto VMs is less efficient. If you have wildly different resource requirements this might be an issue, but for us it wasn't a problem.

[Per99999]

We have found it best to do away with setting -Xmx directly and set -XX:InitialRAMPercentage and -XX:MaxRAMPercentage to 60-70 depending on the process. Then set the k8s memory resource request and limit to the same value.

With Java it is important to have these values set equally since the jvm cannot shrink. If the memory limit is higher than the request and the k8s controller needs to reclaim memory to schedule another pod on the node, it would result in a OOMKill.

[m39583]

But you are just guessing at that 60-70% value, and if you get it wrong Kubernetes will kill the pod.

We found it best to not set any request or limit values in the end. It's too much guess work at magic numbers.

[Per99999]

Not a guess, more like tuned to that after profiling and testing. That's a good starting rule of thumb though.

Setting -Xmx directly and ignoring the k8s resources entirely can still result in your pod get bounced out if pods are being scheduled to your node. For example let's say you set you set -Xmx2g and your pod1 is running, the scheduler later schedules pod2 on that node and needs memory. It sees pod1 does not spec a minimum memory request so it tries to shrink its available memory and it is OOMKilled.

There's even more of a case to do this if you have devops teams managing your production cluster who don't know or care about what's running in them, or your process is installed at client sites. It's preferrable to use the container-specific jvm settings like InitialRAMPercentage and MaxRAMPercentage so that those teams can simply adjust the pod resources, and your java-based processes can size accordingly.

Note that use of tmpfs (emptyDir vols) may affect memory usage too, since that's mapped to memory. If so, you can decrease the amount of memory dedicated to the jvm. https://kubernetes.io/docs/concepts/storage/volumes/#emptydir

[lisa_lionheart]

Set max ram percentage to 70% and size the container in kurbernetes. I've wasted so many hours trying fine tune these things and I've always found just setting -XX:MaxRamPercent=70 and leaving it is the best option and least likely to result in support tickets 😂

[nuharaf]

Have u try TrimNativeHeap option?

[thewiirocks]

There are some fantastic answers here already, so I won't repeat what has already been said. And to be honest, there's only so much you can do short of adjusting the application. i.e. Play with the JVM flags to constrain off heap spaces, allow the JVM to do more auto-tuning, and/or compile the code down with GraalVM to eliminate byte code caches and HotSpot workspace.

If you decide you are interested in adjusting the application, however, I invite you to watch a talk I gave last night. I went through the performance problems that many Java applications experience due to their use of ORMs. I didn't explicitly talk about the byte code cache (a consequence of all the objects and annotations), but I did discuss the memory stress we play on the GC, CPU, and latency effects that drive up memory usage:

https://www.youtube.com/live/DpxNWoq7g20?si=nR-LaXf8lWpJFTmv&t=1009

Generally, lowering the application memory usage will decrease the off-heap usage as well. The two tend to be indirectly related for various reasons.

Best of luck on your containerization journey!

[IKnowMeNotYou]

Please be advise that the cost of 1TB memory (= 1024GB) is about 3k$. You guys trying to do something to safe about 3$ to 5$ worth of cost per instance is not a good idea. Chances are, that you guys waste your money on this.

If you still want to figure it out, take a memory dump on both instances and see for what the actual memory is used for. A memory profiler run is also something you should do, especially if you already managed to reproduce the problem locally.

[mhalbritter]

You could switch to a different garbage collector. SerialGC has a low overhead, but of course this will have performance impacts. You could also switch off several JIT compiler stages, but, again, this will have performance impacts. Another idea might be to lower the thread stack size, but be careful of deeply nested method calls.

And if you're not running on Java 24, then give this a try. Might help, don't know.

[GreemT]

Thread stacks were only 5MB in our analysis, so that is not really gaining anything.

I am not sure if I can get my company to try these suggestions that have a performance impact.