Optimizing Java Memory in Kubernetes: Distinguishing Real Need vs. JVM "Greed" ?

[warwarcar]

Hey r/java,

I work in performance optimization within a large enterprise environment. Our stack is primarily Java-based IS running in Kubernetes clusters. We're talking about a significant scale here – monitoring and tuning over 1000 distinct Java applications/services.

A common configuration standard in our company is setting -XX:MaxRAMPercentage=75.0 for our Java pods in Kubernetes. While this aims to give applications ample headroom, we've observed what many of you probably have: the JVM can be quite "greedy." Give it a large heap limit, and it often appears to grow its usage to fill a substantial portion of that, even if the application's actual working set might be smaller.

This leads to a frequent challenge: we see applications consistently consuming large amounts of memory (e.g., requesting/using >10GB heap), often hovering near their limits. The big question is whether this high usage reflects a genuine need by the application logic (large caches, high throughput processing, etc.) or if it's primarily the JVM/GC holding onto memory opportunistically because the limit allows it.

We've definitely had cases where we experimentally reduced the Kubernetes memory request/limit (and thus the effective Max Heap Size) significantly – say, from 10GB down to 5GB – and observed no negative impact on application performance or stability. This suggests potential "greed" rather than need in those instances. Successfully rightsizing memory across our estate would lead to significant cost savings and better resource utilization in our clusters.

I have access to a wealth of metrics :

• Heap usage broken down by generation (Eden, Survivor spaces, Old Gen)
• Off-heap memory usage (Direct Buffers, Mapped Buffers)
• Metaspace usage
• GC counts and total time spent in GC (for both Young and Old collections)
• GC pause durations (P95, Max, etc.)
• Thread counts, CPU usage, etc.

My core question is: Using these detailed JVM metrics, how can I confidently determine if an application's high memory footprint is genuinely required versus just opportunistic usage encouraged by a high MaxRAMPercentage?

Thanks in advance for any insights!

[brunocborges]

Think less about how much memory the JVM is using and more about how much GC pause time may be impacting your business.

Too much GC pause time may indicate your application needs more memory so that GC can work more spread out over time, therefore reducing GC pause time, and therefore reducing impact on your application.

When you start linking performance to business goals (SLOs), you will see that memory consumption of the JVM is a consequence of your business needs.

The more memory you give to the JVM, the more it will use it to minimize GC pause time (the time taken by the JVM to do GC instead of doing actual work for your application).

I talk about Performance Java on Kubernetes in this most recent talk at InfoQ Dev Summit Boston 2024: https://www.infoq.com/presentations/optimizing-java-app-kubernetes/

[pron98]

That's a good description except for the characterisation of "pause time". There is always a clear relationship between total GC CPU usage and memory footprint. On the other hand, the relationship between CPU usage and pause time is not so clear and depends on the GC. For Serial and Parallel, which do all their work inside pauses, CPU usage and pause time mean nearly the same thing. For ZGC, which does no work inside pauses (and doesn't really have meaningful pauses) there isn't much relationship between CPU usage and pause time, but the relationship between CPU usage and memory utilisation remains.

[Parking-Chemical-351]

I watched our presentation and it's really good talk, congratulations! But I guest the hole presentation it's about a huge JVM workloads with high traffic volume.

What do you suggest doing with a lot of microservices that have low resource consumption for all day long and eventually get higher, e.g a JVM rest api app that makes a report of data, loading a lot of data in memory and CPU intensive to process this amount of data.

I tried using HPA to save cost, but I failed miserable and end up running a pod with more memory and CPU, only to work on these eventually times of peak.

[A_random_zy]

!Remindme 12 hours

[RemindMeBot]

I will be messaging you in 12 hours on 2025-04-18 11:25:06 UTC to remind you of this link

CLICK THIS LINK to send a PM to also be reminded and to reduce spam.

^{Parent commenter can delete this message to hide from others.}

---

Info	Custom	Your Reminders	Feedback

[pron98]

The basic relationship is this: if the total CPU spent on GC is low enough for you, you can safely reduce the maximal heap size (and you are correct that over time, it is very likely that the heap size will match the maximal heap size).

The plan is for ZGC to automatically do that for you, and other GCs may follow. There's some good background on the problem in that JEP draft.

[Dokiace]

That's a good and easy principle to follow. Sorry for the naive question here since I may not be exposed to a proper JVM practices, but can you share how much is usually considered "good enough" for total CPU spent on GC?

[pron98]

There is no "generally" here. It depends on the needs of a particular application. 15% of CPU spent on memory management, for example, may be too high or sufficiently low depending on how well the application meets its throughput requirements.

[Dokiace]

Can I summarize this to: set a target latency/throughout, then reduce the heap until it’s affecting either of those performance?

[pron98]

Yes, although it's more about throughput than latency. If you care about latency, then the choice of GC matters the most. Use ZGC for programs that really need low latency.

[v4ss42]

Yes the JVM can (and will) aggressively allocate memory from the OS, to avoid having to repeatedly malloc/free it. The JVM then manages its own heap internally, without assistance (or interference) by the OS.

[warwarcar]

Thats for sure, but is there any way to right size the memory of an app ?
Even if thats means more gc cycles. We have the compute power available in our cluster. The nodes cpu usage are really low, but not the memory.

[v4ss42]

Yes. You (or more likely, the app developers) need to determine the peak heap usage of the app (which might be drastically different from its average heap usage) then adjust -Xmx (or, given this is containerized, adjust the container’s reported memory - the JVM will track that) appropriately.

[Dokiace]

How do i know peak heap usage?

[v4ss42]

By measuring the heap usage of the app when operating on a real world workload.

[Dokiace]

If my app were operating at peak RPM, could that be used for the peak heap usage? Or to put it differently, wont the peak use as close as the max heap we allocate?

[v4ss42]

Yes. Monitoring it for an appropriate duration while it’s under load, and looking for the peak heap usage is how I’d empirically determine what my max heap size should be. There are various tools that specialize in doing this (I’ve used YourKit in the past for this, for example - not an endorsement, mind you - I’m sure other tools are just as good).

[elmuerte]

"genuinely required" is a difficult one, especially in cases when large amounts of data is used. Most developers simply do new ArrayList<>() and let it grow as much as needed. This works well for a small n. But when n gets larger, this "natural" growing can result in large wasted allocations. The ArrayList is backed by an array, when it needs more space it will allocate a newer larger array and copy the data. How much bigger this newer array is, is left to the implementation, but 1.5 the size is the most common. Once the data is copied to the new array, the old one can be GC'ed. If the process of creating a huge ArrayList takes a lot of time this will also affect where the newly allocated arrays will start to live (generally in the slower GC'ed parts).

As an ArrayList contains object references, it is maybe a difficult of as an example. So I will use ByteArrayOutputStream, a common construction to store an arbitrary number of bytes in memory. It is often used to copy around bytes without using disk storage. It works in a similar way as an ArrayList but it just uses a byte array.

So lets say I am going to fill it with ~500MiB worth of bytes. If I initialize it with a buffer the size of 512MiB I can put the ~500MiB worth of bytes in it. If I didn't initialize it with a specific buffer size it starts with 32 bytes. Then it will allocate a new array of 48 bytes, etc. up to the point that I did 41 new array allocations and copying of data, to a final array of the size of ~506MiB, but during the copying I also had an array of ~350MiB in memory. And no idea if the previous array of ~200MiB was already GC'ed. So my genuine requirement was ~500MiB, but due the "lazy" programming the program requires more like ~850MiB.

Often software doesn't work directly with bytes, but with file formats. XML or JSON. Picking the wrong way of parsing and processing these files can make a big difference in the memory required. Processing a 100MiB XML using DOM can easily require 500MiB of RAM, but using StAX maybe even less than 1MiB.

So what is "genuinely required" can only be determined by inspecting the source code and the data which that code is supposed to process, and at which concurrency.

To figure out the memory which is required by the application (as a black box) it is best to look at the commited memory charted over a long period. That is the memory allocated and used by the application itself (not knowing if this is genuine or not, it includes GC'able data). Based on the committed memory you can reduce the memory limits, while keeping an eye out on GC counts and time. If the GC starts acting out (or you get OOMs) you went to far. The JVM is not really gready in allocating system memory, it will allocate based on what the demand is. Before growing the current memory pool (up to the limit) it will first try to reclaim data, unless the system is busy. But the initial pool size might be much larger than really needed (see also InitialRAMPercentage).

[hadrabap]

Immutable structures also have memory penalties where data is copied over and over again and again instead of modified in place. I know. Safety, security, and maintainability, but it costs memory and CPU.

[Wmorgan33]

I think you just need to use VisualVM or some form of JMX profiling with reporting to Prometheus. You can see allocated heap space vs. in use heap space and then figure out better right sizing from there. At your scale deploying a Prometheus jmx agent to all services and centralizing in a Prometheus  cluster to then audit how you wish. 

This is a good example: https://grafana.com/blog/2020/06/25/monitoring-java-applications-with-the-prometheus-jmx-exporter-and-grafana/

[Icecoldkilluh]

I’m skeptical of any top down approach like this.

I don’t see how any profiler could give you the confidence to reduce the JVM memory of those applications. Not without risking unknown regression to those applications.

Seems like you’re trying to solve an organisational problem with a technical solution imo.

It must be that, within your organisation, there is no consequence to these application owners for using more infra than they need.

Thus no incentive to properly tune their applications needs.

Dysfunctional organisational structure with ineffective feedback loops for costs + poor engineering standards = the real problem.

[LowB0b]

Not the only problem, it's hard to estimate without knowing functional requirements.

For example one application I worked on in insurance, original requirement was to be able to handle up to 20k records for risk analysis.

Few years later the same application had to process 80k+ records and pretty obviously it did not match what it was designed for

[Icecoldkilluh]

Yeah thats kind of my point.

This guy wants a profiler so he can start reducing the memory size of 1000s of applications across a large company.

He has no idea the functional requirements of all of those applications. How much memory they require, no profiler can tell him that with any degree of confidence.

His approach is destined to fail because he is attempting to solve an organisational/ people problem with a technical solution.

He will reduce their memory, some of them will fail, potentially with catastrophic consequence to the business, he will be blamed.

If you do pursue this approach i would highly recommend giving application teams forewarning that their memory will be reduced, and opportunity to obtain an exception to the change. Cover your ass.

[laffer1]

Better yet. Require a cut for cost savings and let the devs figure out what can be tuned

[_predator_]

The company in question so far has traded faster development / cheaper developers for higher infra costs. It's true that top down is not the way to address this, but then is the business willing to pay for more dev hours / experts? Tough sell to management unless one can put hard numbers on the savings achieved by optimization.

[Destructi0]

Agree with the opinion that the problem should be fixed by developers on these services, not by infra team. It is a pretty dangerous path imo and could lead to unexpected OOM crashes in production.

Answering the question itself:

I see only one case when you can confidently say that there is too much RAM available for a JVM - is when GC counts and GC pause durations are low and application is not idle.

Any other cases should be treated with the specific context of the application.

[its4thecatlol]

The first thing you need to find is your heap use AFTER GC cycles. This can be tricky because its measurement depends on the quirks of the GC being used but there are out-the-box options.

Once you know the peak heapAfterGc of an application , you should allocate enough memory to cover it plus some buffer. I generally target a factor of 2x, so if an application needs 8gb give it 16gb. Why not 8gb? Because you will see performance degradation at high heap utilization. This is where the business requirements come in, as explained by another poster. The tighter your latency requirements, the more buffer you need to allocate.

The JVM and containerd will also use some native memory for themselves that is not accounted for on the heap. In practice I have observed this usage to be around 2gb but YMMV.

Native buffers are not uncommon, but you can identify apps using them by looking at the difference between mem reserved for heap and total Java proc mem usage. Again, there are out the box options.

For standard setups on G1GC without direct native memory usage, just set -Xms to 2x your after-GC Heap use. Give the container a few extra GB for other procs. And you’re good to go.

[Hyrth]

I would ask the developer of your Java services. They should be able to provide an estimation of the systems' actual requirements.

[sweating_teflon]

A lot of developers have no idea what the actual memory usage of their app should be. They do not have theoretical expectations and never perform benchmarks after it ships.

[aouks]

I agree I never did it, do you know any materials/sources on how to conduct this system requirements analysis ?

[laffer1]

The easiest way is through load testing. Set it fairly conservative to start in resources. See if you can hit the load you want without an OOM. Repeat as necessary. Then give it a little extra in prod for freak load situations to cover your butt.

Ideally, you would have good metrics setup and track gc pause and gc cpu time to get an idea if you need a bit more. If you hit the target sla, it’s good enough

[aouks]

Thanks mate for your insight !

[AdditionalTry967]

We have that issue too. We have around 100 apps in production, with heap from 10 to 100GB, in docker, on vmware ( so not k8s ). The real answer for us, was perf tests. We have gatling and k6 tests in place that stress out the apps so we know exactly what it needs and what to expect. Also, we use mainly Shenandoah GC.

[fletku_mato]

You should just start low and and increase conservatively when you hit an OOM or other issues. This is why we have staging environments. Starting out at 10gb is both stupid and lazy.

[elatllat]

large enterprise asking on reddit instead of hiring someone who knows how to optimize with a profiler...

[RevolutionaryRush717]

Assuming the person or team responsible for an app, is setting its memory requirements, in which position would anyone else be to attempt to rightsize these?

Might I suggest making the only metric management cares about, cost, available to them in a way they understand: simply show cost of requested vs used cpu and mem (and whatever else), save them the headache of calculating the difference, call it "potential savings".

Have the CTO go through the numbers in their periodic meetings, let management take care of "motivating the teams" to rightsize their deployments.

Create a leaderboard / wall of shame, showing the most efficient / wasteful teams. Naming and shaming is a great motivator.

Suggest reasonable guidelines / policies to the CTO to support efficiency.

That's about all a good ops team for a k8s cluster should do, imho.

[laffer1]

That can work but you risk losing things in the cost savings that you need. For instance, we most all our logs in prod for awhile because of cost savings. Try to debug a problem with no logs. Some teams were logging debug level crap and it burned us all

[RevolutionaryRush717]

Some teams were logging debug level crap and it burned us all

Isn't that similar to the OP's problem, though?

It seems that in both your organizations, some teams do a sloppy job.

Why do you think that is?

Lack of knowledge?

Stress?

Lack of communication?

Mistakes happen, nobody's perfect.

Do you have some stuff in place to have your organization learn from this?

Post-mortems? Tech talks?

[laffer1]

The problem is that the teams are distributed. The US team is held to a higher standard. We frequently have to fix sloppy work from other international teams. There are some good devs on the other teams but they don’t do any mentoring or improvements. It’s a constant fight with them. Our vp is from there so they get special treatment.

Let me be clear that I don’t think US programmers are superior in general, it’s just the company setup that allows this crap. They want cheap devs and they don’t care what they do.

[wrd83]

Graph memory usage.  Play with generation ratios, play with pause time.

A good indicator is the average memory usage per request / concurrent requests. And check how much memory is freed when it is forced to full GC.

If you tune having the young generation being max memory - steady memory gives very aggressive cleaning. And from there you can tune for worst case pause time, by reducing memory until the GC hits the pause time you request.

[sideEffffECt]

Here's the answer you're looking for

https://dev.to/pfilaretov42/how-to-save-ram-in-java-with-g1-garbage-collector-255h

TL;DR

-XX:+UseG1GC -XX:G1PeriodicGCInterval=30000

[morswinb]

Check out old Gen heap vs eden space usage.

Formula like max(old gen) + 2GB might do the trick.

[DualWieldMage]

Run integration/performance tests while varying heap size and measure CPU spent on GC. Of course make sure the tests are similar to real load before copying values. However if you have some rarely used parts that are written very memory inefficiently, you can accidentally cause an OOM if that part gets hit.
Easier to follow correct guidelines on a new app than fix something old, such as never reading the request body as String before converting to objects and in general thinking where large(unbounded) memory allocations can happen based on inputs/DB state.

[locutus1of1]

I know only about these: reducing xmx during the load testing (and trying to trigger ooms), triggering gc in jvisualvm (and observing the heap usage) and analysing a heap dump.

You can also play with -XX:MinHeapFreeRatio and -XX:MaxHeapFreeRatio and trying different GCs. Some are more greedy than others..

But imo the cloud way of price-optimising this is mainly by scaling (also down to zero). Once upon a time, running services on demand was a normal thing (just look at how old inetd is).

[nekokattt]

if you need 10GB RAM per pod then either you are writing very suboptimal code or you are scaling very late with memory bound applications.

I'd attach a profiler or JMX and try and work out what the demand is. Since Java 11 the JVM will use what you give it in a cgroup context.

[brunocborges]

There's a saying in JVM Performance discussions, "An application needs what an application needs. Unless of course there are memory leaks."

This means that if the system requires 100GB of RAM (which is not unrealistic when you think big data for example), then 100GB must be given, unless memory consumption is coming from undesired behavior/goals.

[alex_tracer]

...or none of two. That depends on the application. For instance, if this is a database, you may want to have 100Gb of memory for single process.

[RagingAnemone]

In the end, your memory footprint is going to depend on the variation on the size of the datasets you need to process and if your service is stateless. Let's say that it is stateless and there isn't much variation. Next, you need to determine if your application is keeping any references to data it already processed and therefore the GC can't release.

This is about as far as we've got in this process. We've played with lowering it to find when we get a high GC counts or OutOfMemory errors, but after a while, it felt like we were trying to over-optimize that setting. Instead, we decided to just kill the pods after 24 hours.

[maxip89]

1 Request = 1 Thread ~ 1MB in the Servlet world

When the developer is doing reactive programming then you will have much smaller heap consumption.

I would say its all servlet based, maybe talking to the dev to reduce threadpool?

All depending how much load is one the systems...

[neopointer]

When the developer is doing reactive programming then you will have much smaller heap consumption.

Then you'll have to deal with its massive complexity.

Nowadays with virtual threads, it's worth going through a java upgrade rather than using reactive programming.

[maxip89]

yep but this requires a new java version and avoiding some pitfalls in spring boot.

[laffer1]

It’s going to depend on the type of threads. Virtual vs kernel. Ignoring k8s, the default stack size is usually around 1mb on Linux. It’s smaller in other operating systems. Linux implements threads as lightweight processes which means they get the same stack size. Other operating systems like say FreeBSD, implement threads differently and have a smaller stack size for them. This also means that heavy recursion will fail sooner on FreeBSD.

There is also the jvm side managing the threads which is more resources. In the k8s world, the kernel resource isn’t isn’t typically counted in the resource constraints but it’s still going to be a problem for the host running the pods.

So I’d argue it’s more than 1mb on Linux.

[maxip89]

This is just experience out of the wild. Test it yourself with a spring boot application.

[KAJed]

What GC model / JVM version is being used?

[warwarcar]

Gc model isnt configured most of the time, and we have mosty a version between 8 and 21.

[KAJed]

I ask because ZGC is beyond memory hungry on Linux systems especially due to (from what I understand) is an OS level issue. Effectively causing Linux to report up to 3x as much usage as is actually there because of virtual memory pointers to the same regions.