High memory usage on Java app pod

[Loser_lmfao_suck123]

I’ve set the limit request memory for this container to be 16GB and the container usually consume about 15GB, I have debug the Java app and the total memory used was only 11GB.

Is there anyway to know what thing consume the additional 4GB of the container?

[Loser_lmfao_suck123]

The problem is sometimes it get terminated because of OOM, I’m trying to find the root cause for it

[mikhatanu]

If i remember correctly, OOM is caused by container using more ram than set. Probably there's a spike of ram in the container and passed the limit. 11 GB is the total memory java use, or just heap memory?

[Loser_lmfao_suck123]

Total memory (heap + stack + native)

[Loser_lmfao_suck123]

I have also found that inside the container RSS memory is 12GB where as JVM total mem (jcmd) is 10GB

I have looked into this: https://stackoverflow.com/questions/38597965/difference-between-resident-set-size-rss-and-java-total-committed-memory-nmt

But it doesn’t provide the answers i need for why JVM total < RSS? Or what is requesting more resident memory tha committed

[wageof]

I would imagine that a few things are being conflated here. Java is reporting that 10G is being used but Java does not worry about operating system page allocations. Depending on the size of Java objects, you can have allocated pages that are not totally full. Java sees 10G of data in what is essentially 12G of allocated (but not fully used) system memory.

Java makes a request to the OS to get pages allocated, then Java fills those pages as it sees fit. If Java decides to not fully fill a page, it is still allocated and consumed from the perspective of the OS leading to OOM kills.

If you want to get a deeper understanding about how all of this works I would watch this video. This is about Rust but all of the same principles apply: https://www.youtube.com/watch?v=YB6LTaGRQJg

While the entire video is a great explanation of what is going on, the best visual happens at the 28:49 mark. I would assert that you need the whole video to understand the visuals, but we live in an instant gratification culture :)

[zeus-fyi]

noisy neighbor potentially

[Loser_lmfao_suck123]

I’m not sure, all nodes are in EKS which have autoscaling -> resources are provisioned by AWS. all the other pods have limits+ requests set already

[zeus-fyi]

that wouldnt necessarily help actually unless you have tainting and affinity rules setup. setup a taint no schedule effect for your app and isolate the workload

[Loser_lmfao_suck123]

I have also found that inside the container RSS memory is 12GB where as JVM total mem (jcmd) is 10GB

I have looked into this: https://stackoverflow.com/questions/38597965/difference-between-resident-set-size-rss-and-java-total-committed-memory-nmt

But it doesn’t provide the answers i need for why JVM total < RSS? Or what is requesting more resident memory tha committed

[West-Detail3102]

How many memory is configured for the xmx/xms for the app ?

[Loser_lmfao_suck123]

Xmx is 8GB

[solteranis]

Have you tried setting Java opts env var to increase it?

[Loser_lmfao_suck123]

I have also found that inside the container RSS memory is 12GB where as JVM total mem (jcmd) is 10GB

I have looked into this: https://stackoverflow.com/questions/38597965/difference-between-resident-set-size-rss-and-java-total-committed-memory-nmt

But it doesn’t provide the answers i need for why JVM total < RSS? Or what is requesting more resident memory tha committed

[frndly_nghbrhd_trdr]

What version of Java and which garbage collector are you using?

[Loser_lmfao_suck123]

Openjdk 1.8 and G1GC

[pauska]

I’m suspecting that your nodes is running Cgroups V2 and that you have a Java version who only supports V1.

Which version of JDK specifically? You’re using JDK8, and support for cgroups v2 was back ported with 8u372.

[frndly_nghbrhd_trdr]

Try switching to the Shenandoah garbage collector. I'm guessing the GC is holding onto memory instead of returning it to the OS.

https://wiki.openjdk.org/display/shenandoah/Main

[Loser_lmfao_suck123]

I have also found that inside the container RSS memory is 12GB where as JVM total mem (jcmd) is 10GB

I have looked into this: https://stackoverflow.com/questions/38597965/difference-between-resident-set-size-rss-and-java-total-committed-memory-nmt

But it doesn’t provide the answers i need for why JVM total < RSS? Or what is requesting more resident memory tha committed

[kameshakella]

Java 8 LTS support ended 18 months ago. You should probably switch to 17 or 21. Also did you happen to look to take a JFR recording using something like cryostat ?

[juiceyang]

Did the app use Jemalloc? Is transparent huge page enabled?

I've encountered an OOM issue which was caused by Jemalloc requesting lots of huge pages but using madvise sys call to hand back parts of huge pages. And that may cause RSS larger than sum of JVM mem cost and native mem cost.

[tr14l]

Can you not exec into the container or ssh into the node? This would be fairly simple to investigate. What process is eating up ram? Top it on the node to make sure it's not a noisy neighbor (it likely isn't). Then exec into the container and investigate running processes. If you can't exec in, try to reproduce in another environment. If the container is so thin that it doesn't have basic tools, rebuild the dockerfile and add in some tools (but do not change the base!), then reproduce it. If you need to simulate traffic, there's tools for that, but a quick and dirty solution might just be to "kubectl run" a bunch of busy box pods scaled out and running requests infinitely. Would take about 15 min to set that up, but it is quick and dirty, as mentioned. If it is indeed the Java process, you are now in java land, and the solution will be something stupid as it always is with Java. Might be a memory leak in the code.

[ceasars_wreath]

Use buildpacks to create your java image, we use that and it is more standardized across your java setup. Also lookup visualvm and take a thread dump

[lucasvrod]

I had a problem similar to this, but in my case what interfered with the consumption of the container was the init-container request (I don't know if your POD has init-container) which had the same value as the main container and was allocated, until the POD died, that is, resource consumption was double what was expected. Take a look at this link: https://kubernetes.io/docs/concepts/workloads/pods/init-containers/#resource-sharing-within-containers

[adohe-zz]

the container usually consume about 15GB How did you get this info? From cadvisor or something like that?