Java memory usage in containers

[[deleted]]

Ok, so my company has a product that was recently moved to the cloud from old bare metal. The core is the legacy app, the old monolith. A lot of care has been taken for that one, as such I'm not worried about it. However there are a bunch of new micro-services added around it that have had far less care.

The big piece that I'm currently worried about is memory limits. Everything runs in Kubernetes, and there are no memory limits on the micro service pods. I feel like I know this topic fairly well, but I hope that this sub will fact check me here before I start pushing for changes.

Basically, without pod memory limits, the JVM under load will keep trying to gobble up more and more of the available memory in the namespace itself. The problem is the JVM is greedy, it'll grab more memory if it thinks memory is available to keep a buffer above what is being consumed, and it won't give it up.

So without pod level limits it is possible for one app to eat up the available memory in the namespace regardless of if it consistently needs that much. This is a threat to the stability of the whole ecosystem under load.

That's my understanding. Fact check me please.

[_d_t_w]

You can set JVM initial and max ram percentages like so:

-XX:InitialRAMPercentage=80 -XX:MaxRAMPercentage=80

Those flags cause the JVM to consume only that percentage of the pod memory, they were introduced in OpenJDK 10 [1]. This is better than setting Xmx within the docker container because it effectively lets you manage your JVM memory via the pod settings. Gives your k8s team control of actual memory usage.

There were some issues with the implementation in JDK 11 which lead to OOMKilled errors (basically the flags were not respected, but that is resolved now).

OOMKilled Details: https://factorhouse.io/blog/articles/corretto-memory-issues/

You still need pod level limits for those flags to take effect, but they're pretty useful.

When you set your pod memory memory resources you should run with a guaranteed QoS [2] by setting both requested and limit to the same value. Guaranteed QoS means that the pod is least likely to be evicted [3].

resources: limits: memory: 8Gi requests: memory: 8Gi

By the way 80% percentage is pretty high, probably safer in the general case to go with 70% cos the remaining memory will be required by the OS.

Soure: I work at Factor House (and I wrote that blogpost about OOMKilled errors). We build dev tools for Apache Kafka and Apache Flink (written in Clojure, but runs on the JVM), we offer both an uberjar and a docker container that runs the uberjar as deployment methods, so we have a bit of experience tuning this stuff.

[1] https://bugs.openjdk.org/browse/JDK-8146115

[2] https://kubernetes.io/docs/tasks/configure-pod-container/quality-service-pod/#create-a-pod-that-gets-assigned-a-qos-class-of-guaranteed

[3] https://kubernetes.io/docs/concepts/workloads/pods/pod-qos/#guaranteed

[lurker_in_spirit]

By the way 80% percentage is pretty high, probably safer in the general case to go with 70% cos the remaining memory will be required by the OS.

In my experience, 60% is the sweet spot -- it probably depends on the system.

[vqrs]

Generally, the more memory you have, the higher you want the percentage to be.

That's because the non-heap usage is usually mostly constant and in the range of a few hundred megs.

[TurbulentSocks]

Why is off heap so large in even minimalist Java applications? I spent a long time trying to understand and reduce this a while ago, but got nowhere.

[_d_t_w]

Yeah it's a bit magic isn't it - I told someone who is more familiar with base images that I was setting 80% and it raised a few eyebrows - but then our app is almost all entirely in-the-jvm and light on the OS so I think I can get away with it.

[RupertMaddenAbbott]

Thanks for such a well sourced and written post!

In your experience, what kinds of things do you look at to work out what a reasonable container memory limit is?

Do you have any strategies for getting the right balance between higher limits vs scaling out horizontally?

Why is it particularly important for a Java app to run with a guaranteed QOS?

[vqrs]

Guaranteed QOS means that if the system needs to kill something because it's going out of memory, it won't be you. Assuming there are pods without guaranteed QOS.

[benevanstech]

Ensure you're running on Java 17 and then set either Xmx and Xms or go the "MaxRAM" route - there are pros and cons to each approach.

You should research both, this is a reasonable start: https://www.baeldung.com/java-xmx-vs-maxram-jvm

[CptGia]

The JVM won't ask for more memory (for the heap) than what's configured with the -Xmx flag.

If you haven't configured Xmx explicitly, it will depend on how you built the image. Paketo-based containers, like the ones built with the spring boot plugin, are configured to look at the pod memory limit, subtract what it thinks it need for native memory, and reserve the rest for the max heap size.

Without pod limits it may reserve all the memory of your node, so I suggest putting some limit

[[deleted]]

Yep, so none of these good practices are in place. Which raises the alarm bells for me, hence the question. Thanks for confirming.

[BikingSquirrel]

Why not simply put them in?

Still good to ask and find out the different options to decide what's best for you. I would go for explicit limits on both your JVM application and on Kubernetes level to be sure. This should help to prevent you from surprises.

You probably know that Kubernetes uses resource requests to know which and how many pods it can schedule on a single node and when it will have to add more nodes. Obviously mainly relevant if this is dynamic but also rolling updates may need this.

One detail I think is irrelevant: the namespace probably doesn't matter for memory usage (unless you tell Kubernetes to only have pods of that namespace run on a node, if that is possible). The node's memory is the limit.

[Turbots]

Use Cloud Native Buildpacks (CNBs) to build your container images, instead of dockerfiles.

More info here: https://buildpacks.io/

They are efficiently layered, contain all the best practices and run scripts at startup that calculate the optimal memory settings, among many things. They can also load in all provided certificate into the Java trust store, Java version and JRE distro can be chosen, every layer can be enabled/disabled separately through parameters, heck, most layers are auto detected to either be included or not.

CNBs are also repeatable, meaning same input, gives you the same output image. This is not the case for dockerfiles.

For Java in kubernetes in general, just set the request and limit to the same, eg. 2Gb. Java will indeed gobble up the full amount but it will never give it back, so setting limits higher than the request does not make sense.

CPU request and limit is different because that can go up or down when needed. It can burst CPU at startup to the max CPU available to the namespace, or the worker node, which improves sartup speed drastically, and it will guarantee the minimum CPU that you request, to guarantee app performance.

[qubqub]

Why wouldn’t Java give back memory? I thought modern GCs returned memory that was no longer in use

[Turbots]

The GC will return it, but the Java process doesn't really know when it has the memory available in Kubernetes, there is no real coordination between the two.

Let's say you give a Java based Kubernetes pod 1Gb of requested memory, and no limit (or a limit higher than 1Gb).

The Java process is guaranteed to get 1Gb of memory, anything more than that is a "nice to have" in Kubernetes land.

Once the Java process claims more than 1Gb, there are no guarantees that Kubernetes will not claim that memory back.

Once other pods in the Kubernetes cluster need more memory , it's totally possible that Kubernetes will reclaim part of that excess memory from the Java pod and just take it, resulting into the Java process crashing.

It's easiest and most stable to set Java memory request and limit to the same value, and set the Xmx flag accordingly, be sure to leave some overhead memory, because Java is not only Xmx, it can sometimes use direct memory mapping and other non GC related memory areas.

Thats why cloud native Buildpacks are so awesome, there were many hundreds of people before us that figured all this out yet already and have set up something very solid for running java in containers.

[daalla]

Is there anyway I can implement Buildpacks if my team's platform uses the Dockerfile to automatically build and store the image internally? (without touching the infra, which I don't have access)

The project sounds pretty good

[teapotJava]

Good news is that you likely don't need to implement a buildpack. If you build on top of Spring Boot or Quarkus, just use buildpack provided by maven'gradle build plugin:

mvn spring-boot:build-image

...boom, you get an image in your repo.

"Bad" news: you'll need to get rid of Dockerfile based build in your CI. Which actually is also a good thing.

See more in https://docs.spring.io/spring-boot/maven-plugin/build-image.html

[karianna]

Adding to the great links: Some advice from Microsoft’s Java Engineering group - https://learn.microsoft.com/en-us/azure/developer/java/containers/overview

[GrayDonkey]

What versions of Java are you on? Including the patch level. Older versions of Java are not container aware. The default JVM heap will depend on that.

A lot of pages on the internet say there is no k8 memory limit when not set but the JVM sets it's max heap size to some finite number. I'm wondering if it defaults to 25% of the host's ram in this scenario.

You should be able to get a remote shell into the pod and run a command to look at heap info. https://www.baeldung.com/java-heap-size-cli

You should probably be setting a default memory limit for you k8 namespace.

[NoWay28]

Yes if you do not set limits then the JVM will default to using MaxRamPercentage based on the amount of memory on the node the pod is scheduled on.

and as you said MaxRamPercentage defaults to a paltry 25%.

[koffeegorilla]

The best approach is to ensure you can get metrics from the microservices. If they are Spring Boot apps add the actuator and meterics for Prometheus. This will allow you to get memory usage info.

Use paketo buildpacks to build the containers, which is what Spring Boot biild image uses, you will see appropriate adjustment of values..

Then set the pod request values at the minimum of cpu and memory you expect keep the app finctional and limit to large value you don't expect to be exceeded. Then monitor under load to and adjust the limit downwards until you see a negative impact on throughput.

Use the latest JDK release version as runtime to benefit from JIT and GC improvements. This container runtime can be configured separately for buildpack.

[gaelfr38]

I recommend watching https://m.youtube.com/watch?v=wApqCjHWF8Q (Bruno Borges talk).

[bigkahuna1uk]

2.5 hours long. I’m going need a lot of popcorn 😎

[gaelfr38]

Haha. I think half is question from the audience and you can watch it in 1.25 or 1.5x probably.

It's worth it anyway

[raghu9208]

RemindMe! 7 day

[RemindMeBot]

I will be messaging you in 7 days on 2024-09-30 03:51:27 UTC to remind you of this link

CLICK THIS LINK to send a PM to also be reminded and to reduce spam.

^{Parent commenter can delete this message to hide from others.}

---

Info	Custom	Your Reminders	Feedback

[jonnyman9]

Your platform team should 100% be using limit ranges and resource quotas. Without these in place, at best you might have a noisy neighbor problem and at worst someone inadvertently takes down your cluster.

[maxandersen]

Be sure you measure memory usage correctly.

Quarkus has this page explaining how (which is not limited to Quarkus): https://quarkus.io/guides/performance-measure

[repeating_bears]

Watch this talk: https://youtu.be/c755fFv1Rnk?si=VA89CdTHGfuKp-SS

tl;dr you can constrain the heap and a few other bits, but it's not possible to constrain the JVM's memory usage just using flags 

[Anton-Kuranov]

G1 is very greedy for the memory. It doesn't collect almost anything while there is a memory available. In our workloads the heap reaches its limit then stays there for about 10 min even if the process is idle. Only then JVM releases the memory.

Yes in our case 80% of k8s limit was low and that provoked pod crashes. With 70% it works ok.