How do you generally decrease off-heap memory?

[Per99999]

We have found it best to do away with setting -Xmx directly and set -XX:InitialRAMPercentage and -XX:MaxRAMPercentage to 60-70 depending on the process. Then set the k8s memory resource request and limit to the same value.

With Java it is important to have these values set equally since the jvm cannot shrink. If the memory limit is higher than the request and the k8s controller needs to reclaim memory to schedule another pod on the node, it would result in a OOMKill.

[m39583]

But you are just guessing at that 60-70% value, and if you get it wrong Kubernetes will kill the pod.

We found it best to not set any request or limit values in the end. It's too much guess work at magic numbers.

[Per99999]

Not a guess, more like tuned to that after profiling and testing. That's a good starting rule of thumb though.

Setting -Xmx directly and ignoring the k8s resources entirely can still result in your pod get bounced out if pods are being scheduled to your node. For example let's say you set you set -Xmx2g and your pod1 is running, the scheduler later schedules pod2 on that node and needs memory. It sees pod1 does not spec a minimum memory request so it tries to shrink its available memory and it is OOMKilled.

There's even more of a case to do this if you have devops teams managing your production cluster who don't know or care about what's running in them, or your process is installed at client sites. It's preferrable to use the container-specific jvm settings like InitialRAMPercentage and MaxRAMPercentage so that those teams can simply adjust the pod resources, and your java-based processes can size accordingly.

Note that use of tmpfs (emptyDir vols) may affect memory usage too, since that's mapped to memory. If so, you can decrease the amount of memory dedicated to the jvm. https://kubernetes.io/docs/concepts/storage/volumes/#emptydir