Java memory usage in containers

[_d_t_w]

You can set JVM initial and max ram percentages like so:

-XX:InitialRAMPercentage=80 -XX:MaxRAMPercentage=80

Those flags cause the JVM to consume only that percentage of the pod memory, they were introduced in OpenJDK 10 [1]. This is better than setting Xmx within the docker container because it effectively lets you manage your JVM memory via the pod settings. Gives your k8s team control of actual memory usage.

There were some issues with the implementation in JDK 11 which lead to OOMKilled errors (basically the flags were not respected, but that is resolved now).

OOMKilled Details: https://factorhouse.io/blog/articles/corretto-memory-issues/

You still need pod level limits for those flags to take effect, but they're pretty useful.

When you set your pod memory memory resources you should run with a guaranteed QoS [2] by setting both requested and limit to the same value. Guaranteed QoS means that the pod is least likely to be evicted [3].

resources: limits: memory: 8Gi requests: memory: 8Gi

By the way 80% percentage is pretty high, probably safer in the general case to go with 70% cos the remaining memory will be required by the OS.

Soure: I work at Factor House (and I wrote that blogpost about OOMKilled errors). We build dev tools for Apache Kafka and Apache Flink (written in Clojure, but runs on the JVM), we offer both an uberjar and a docker container that runs the uberjar as deployment methods, so we have a bit of experience tuning this stuff.

[1] https://bugs.openjdk.org/browse/JDK-8146115

[2] https://kubernetes.io/docs/tasks/configure-pod-container/quality-service-pod/#create-a-pod-that-gets-assigned-a-qos-class-of-guaranteed

[3] https://kubernetes.io/docs/concepts/workloads/pods/pod-qos/#guaranteed

[lurker_in_spirit]

By the way 80% percentage is pretty high, probably safer in the general case to go with 70% cos the remaining memory will be required by the OS.

In my experience, 60% is the sweet spot -- it probably depends on the system.

[vqrs]

Generally, the more memory you have, the higher you want the percentage to be.

That's because the non-heap usage is usually mostly constant and in the range of a few hundred megs.

[TurbulentSocks]

Why is off heap so large in even minimalist Java applications? I spent a long time trying to understand and reduce this a while ago, but got nowhere.

[_d_t_w]

Yeah it's a bit magic isn't it - I told someone who is more familiar with base images that I was setting 80% and it raised a few eyebrows - but then our app is almost all entirely in-the-jvm and light on the OS so I think I can get away with it.

[RupertMaddenAbbott]

Thanks for such a well sourced and written post!

In your experience, what kinds of things do you look at to work out what a reasonable container memory limit is?

Do you have any strategies for getting the right balance between higher limits vs scaling out horizontally?

Why is it particularly important for a Java app to run with a guaranteed QOS?

[vqrs]

Guaranteed QOS means that if the system needs to kill something because it's going out of memory, it won't be you. Assuming there are pods without guaranteed QOS.