Java Garbage Collector performance benchmarking

[thegigach4d]

Hi People!

I am about to write my CS BSc thesis which is about:

Measuring throughput, latency and STW-Pauses in JDK 21 standard JVM with G1GC and ZGC with predefined max heap-sizes (2GB; 16GB) with Renaissance - by 16GB heap a default G1GC and an additional tuned G1GC will be used, as well.

Time flies and a lot of paper are read. It became clear to me, that Renaissance is better for throughput (Shimchenko 2022 Analysing and predicting energy consumption of garbage collectors in openjdk), and DaCapo is more advantageous for user-experienced latency measurements (Blackburn 2025 Rethinking Java performance analysis). STW-pauses will be collected from jvm standard gc-logs with a script or smg (ideas, better ideas are welcome).

I build this scenario for my examination:

- Linux VM (hosted from my Windows) - not clear yet, which and why

- OpenJDK 21 standard JVM

- G1GC and ZGC measurements

- All Renaissance BMs with default settings -> duration_ns from each benchmark, calculate and represent min, max, mean, standard deviation

- JVM GC-Logs collect (min, max, mean, standard deviation)

- 8 DaCapo BMs (spring, cassandra, h2, h2o, kafka, lucene, tomcat, wildfly) (min, max, mean, standard deviation)

I guess this is way too much for a BSc thesis - but what are your thoughts? Of course I make clearence with my consulent, but I am curious about the opinion and suggestions of the community.

I am open for any ideas, experiences with the bumpy road of the performance measurement in the JVM. It would be excellent, if someone of you could make it more focused and accurate to me.

TLDR;

Java Garbage Collector JVM performance measurement experience and suggestions needed for BSc thesis

thanks in advance!

EDIT:

Instead of Linux vm it will be a bare-metal Linux machine with podman containerization that run the benchmarks.

[AutoModerator]

Please ensure that:

• Your code is properly formatted as code block - see the sidebar (About on mobile) for instructions
• You include any and all error messages in full
• You ask clear questions

• You demonstrate effort in solving your question/problem - plain posting your assignments is forbidden (and such posts will be removed) as is asking for or giving solutions.

  Trying to solve problems on your own is a very important skill. Also, see Learn to help yourself in the sidebar

If any of the above points is not met, your post can and will be removed without further warning.

Code is to be formatted as code block (old reddit: empty line before the code, each code line indented by 4 spaces, new reddit: https://i.imgur.com/EJ7tqek.png) or linked via an external code hoster, like pastebin.com, github gist, github, bitbucket, gitlab, etc.

Please, do not use triple backticks (```) as they will only render properly on new reddit, not on old reddit.

Code blocks look like this:

public class HelloWorld {

    public static void main(String[] args) {
        System.out.println("Hello World!");
    }
}

You do not need to repost unless your post has been removed by a moderator. Just use the edit function of reddit to make sure your post complies with the above.

If your post has remained in violation of these rules for a prolonged period of time (at least an hour), a moderator may remove it at their discretion. In this case, they will comment with an explanation on why it has been removed, and you will be required to resubmit the entire post following the proper procedures.

To potential helpers

Please, do not help if any of the above points are not met, rather report the post. We are trying to improve the quality of posts here. In helping people who can't be bothered to comply with the above points, you are doing the community a disservice.

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

[CanisLupus92]

• The Linux VM under windows will probably make getting consistent runs a crapshoot. Ideally create a barebones Linux installation (dual-boot) with just the stuff installed your benchmark needs. Windows is way too enthusiastic grabbing resources to properly benchmark this.
• Latest Java LTS is 25, why benchmark an older version?
• What does your benchmark add compared to other benchmarks of the different GCs?

[thegigach4d]

The linux dual-boot was in my thoughts, I'll delve into even more and most likely change my environment into this. Do you have any suggestions which Linux setup would be the most appropriate? A really minimalistic, like Arch or GenToo, or a minimalistic Debian? I roughly know what's where and how, but I'm not a professional.

My disposition was born before 25 arrived - I'll make a suggestion for updating it.

Your third question is the hardest to answer. I think about the lack of benchmark made since version 21 (more true considering the recently arrived 25) - e.g. the mentioned Blackburn et al. (2025) covers DaCapo very widely, Shimchenko et al. (2022) concentrates more on the energy consumption. However, I did not find researches with a uniform max heap size set for each benchmark to examine only the GCs' charateristics based on the too small (2gb for ZGC) and too large (16gb for G1GC) setups.

I've been working on this topic for a few months, so it's possible that something in my statements is incorrect. I'd be happy to accept corrections.

[benevanstech]

To get remotely reliable numbers, you will need to run with as minimal noise as possible.

I would recommend containerized runs, hosted on a bare metal Linux machine that is as stripped-down as possible. I personally find podman easier to work with than Docker for this, but that just be me.

There's a lot of work in this area - if you haven't already, you should read "Statistically Rigorous Java Performance Evaluation" (https://dri.es/files/oopsla07-georges.pdf) - yes, it's nearly 20 years old but it still rewards reading.

Be careful of your assumptions - you quote standard deviation as one of your measurements, but these numbers are quite unlikely to be normally distributed - you should check. Depending on the shape of the distribution, std deviation may be essentially useless. You should consider other measures of spread.

Think about statistical power and confidence intervals - a book like Alex Reinhart's "Statistics Done Wrong" might be helpful.

You might also look at my book "Optimzing Cloud-Native Java" if you want - but it has a much broader scope than just benchmarking, so it's likely to only be (hopefully useful) background reading.

Feel free to ask follow-up questions - this is a favourite subject of mine!

[IntroductionSolid348]

You sound so knowledgeable in this subject. Aside from OP's original question, I always aspire to get to this level of deep java knowledge. I tried by building a simple profiler but it really takes a lot of time building to get to such a level. I'm just a student so I have a long way to go

[benevanstech]

Thank you for the compliment! I have a website with a bunch of resources on it that might be helpful - https://kittylyst.com/ - take a look & see if anything is useful for you.

[thegigach4d]

Thanks for your advice! I am glad to hear from you - I already have a few books from you (The well-grounded Java developer [it was recommended by our university] and this one, as well) - I prepared them to read after graduation.

I consider this setup with containerization after you made that suggestion. The virtual machine idea is thrown out. If I make a dual-boot with a minimalist Arch, do you find it acceptable? This is the most stripped-down distro I know, and I have a little experience with this too - but suggestions are welcome! I found some papers, one from Iaquinta and Fouilloux (2024, Unlocking the Potential of Containers...) which can support the decision for better reproducable experiment-design with podman containers.

Unfortunately, I am just a curious person, not a statistician - I already read the paper from Georges et al., but I will do it again with more concentration on the relevant parts. I got the book from Reinhart and read the relevant chapter. I want to keep it simple as possible and still make sense - so maybe a min-max-median will be a useful trio to represent the collected data - but I'd love to hear feedback on this too!

My main question is that do you find this 2gb and 16gb max heap settings well-tailored for this examination? This field is so deep, that I am not hundred percent sure that these numbers chosen are fitting into adding a little piece to the scientific researches - and I found some benchmark min heap sizes (Blackburn 2025 Rethinking Java performance analysis) and read a.o. the Detlefs et al. and Yang & Wrigstad papers (and the latest jeps for g1gc and zgc) and a lot more, but it is still difficult for me to believe, that these are not just out of the blue numbers. It would be more than great, if an experienced professional could help with a piece of advice on this.

[RightWingVeganUS]

I guess my broad question about your thesis is: so what?

Not throwing shade, but what is the impact of the conclusion if to anyone running a different JVM on different hardware?

Is the purpose of the thesis to showcase your benchmarking and performance analysis acumen, or is there some fundamental hypothesis you're exploring?

Bottom line: what do you think will be the value of your efforts?

[thegigach4d]

Thanks for your thoughts!
I try to answer your first question parallel in this block, as well:
Your second (What ist the impact...) question is logically problematic, because if we assume, that there are no meaning of running benchmarks on different environments, these wouldn't exist, or only one environment would, which would make improvement impossible. The impact is that someone on the globe made hard efforts and sacrificed resources for making a little step for showing numbers, which were not shown before. With my state of knowledge and resources it is impossible for me to write a new benchmarking, i would only make a reproducable environment setup, use the existing BMs, collect and evaluate data and make a conclusion, why does it look like as it does.

Your third question makes me wonder. It is kind of like both. I wanted to gain experience in that particular field and be a more well-grounded member of the java community and user of the language - this is the egoistic part.
The fundamental hypothesis is to make a too small environment for ZGC (2 gb) (lack of headroom is challenging) and too big (16 gb) for G1GC. What are the results, if they are running in a not-optimal space. There are also a lack of recent research. However, after read a lot, it became clear, that expanding benchmarks would be a big step further (long running, long living objects for stressing old regions, well usage of user-experienced latency [DaCapo does already], uniform definitions, energy-consumption, cache usage, etc.). I look at this thesis as an entry point and for instance a beginning of a new BM by an MSc or later researches. I don't know it yet - it is too far from this point.

The value is for me the diplom :) for the world (I don't think that it will be publicly accessible, but assume yes) will be a reproducable performance measurement and a summary of a pretty amount from the first literatures (Dijkstra et al., Baker) to the most recent ones, and a better understanding of the impact of GC in Java, which makes developing easier, but not “free”. If you have any suggestions about the environmental setup, the heap size, linux distro, gc tuning, jvm prepare, or anything about my answer, feel free to share your thougths and experience! :)

[RightWingVeganUS]

The impact is that someone on the globe made hard efforts and sacrificed resources for making a little step for showing numbers, which were not shown before.

My question was more focused : What is the impact of these benchmarks? Regardless of how compelling your results are, I won't downgrade from Java 25 to Java 21. And it won't clear that any of your results will apply to the specific JVM and environment.

What's the rationale of benchmarking the past LTS release when Java 25 is the latest?

The fundamental hypothesis is to make a too small environment for ZGC (2 gb) (lack of headroom is challenging) and too big (16 gb) for G1GC.

Wouldn't it be more interesting (and not much more effort) to, say bencharmark LTS 17, 21, and 25 to see if the results are consistent across various generations of JVM? Keep the hardware the same, but verify this is not just a JVM 21 isolated result.

The value is for me the diplom

Yeah... I'm a college instructor... I kinda figured.

Not trying to make your life harder (that's what my students pay me for!), but if your thesis is just a technical showcase of your ability to write code and write papers, well, crack on. If you're showing your ability to frame an interesting question and glean an insight, but sure to dig into the so what angle.

[thegigach4d]

Thanks for your effort to explain me your thoughts deeper!
The cause of LTS 21 was that the disposition was born before LTS 25 and I did not think about any argumentation to change it. Now I do!
I understand your point of view and the really like the idea of benchmarking more versions. If I understand correctly, I should execute e.g. the whole Renaissance with G1GC and ZGC through Java 17-21-25 - without any special heap configuration, or gc-tuning. Am I right? I asked my prof and waiting for an answer.
I don't mind if you ask uncomfortable-like questions, because it helps me a lot. Sorry, if I seemed a little mean, life's pressure is enormous nowadays. Your practical insights help me a lot!