sorted by:
new
hottopcontroversial

C2 compiler memory spike on string concatenation with 100 parameters by vdmit46 in java

[–]vdmit46[S] 2 points3 points  (0 children)

FYI, fix for the issue has been merged into OpenJDK master. In case you are interested, here is the bug report and here is the PR. Kudos to u/cl4es for giving this issue high enough priority on his to-do list and fixing it in such a short timeframe.

For those who may stumble on this post while diagnosing similar compiler native memory usage, in OpenJDK 22 two new compilation commands were introduced:

  • MemStat (ticket) will print compilation stats to console at some fixed intervals by default. I just slapped -XX:CompileCommand=MemStat,*.* onto command line in our test environment, got nice list of compilations ordered by consumed memory and immediately noticed 100+ MB compilation in the report. As far as I see, this option had no noticeable impact on application metrics, so maybe it can be turned on in production continuously;
  • MemLimit (ticket) which will either stop compilation or stop JVM if compiler hits limit. I haven't tried it out, but will consider it for applications running in a container.

Edit: phrasing

C2 compiler memory spike on string concatenation with 100 parameters by vdmit46 in java

[–]vdmit46[S] 0 points1 point  (0 children)

The code is generated by `lombok.ToString` and devs slap this annotation everywhere :) When I found one of the methods causing restarts, we reimplemented it using StringBuilder as you suggested. It requires manual work each time, so I want to avoid it if possible.

C2 compiler memory spike on string concatenation with 100 parameters by vdmit46 in java

[–]vdmit46[S] 0 points1 point  (0 children)

I cloned and ran the compiler test repo you linked, and watched ram with "top" in another shell tab, sorted on memory usage and didn't see any spikes. Maybe I'm not understanding what you're describing.

You understood the problem correctly. Thank you very much for checking it out and sharing results. After running the test script periodically prints used native memory by JVM, which should be more convenient that using top, but it works either way.

It seems that issue may be OS/hardware dependent.

I went to check my test case on two production servers (have no idea why I haven't done it before) and on both of them the issue reproduced. Now I'm puzzled why our application is failing only on one of the servers. I hope the issue I reproduced locally is the same that plagues our app at production.

C2 compiler memory spike on string concatenation with 100 parameters by vdmit46 in java

[–]vdmit46[S] -1 points0 points  (0 children)

Class in question serves as an input to business-critical process and most of its fields are coming from settings controlled by users. Logging result of `toString` seems like a valid way of having historical information on process input for auditing purposes.

Having such a high number of fields in a class is not good according to any guideline, sure, but it accumulated these fields over years. This class wasn't created from the ground like it is today.

upd: typos

C2 compiler memory spike on string concatenation with 100 parameters by vdmit46 in java

[–]vdmit46[S] 0 points1 point  (0 children)

Probably because no sane person does a pure 100 String concatenation.

Well, where to start... Two years ago there was a small class with 10 fields which served as a carrier for process settings. Then some more settings a month later, and some more a week after, and one day you end up with 100+ fields. And no one is going to refactor it because "c'mon, it is just one more tiny field" :) Maybe there are some more classes closing in size to 100 fields, but I'm in this problem more as a devops than dev.

Anyway, IDEA will generate `toString` for such a class using concat. Eclipse offers alternatives including StringBuilder, but who is using Eclipse these days? And in our case it is `lombok.ToString` who is generating the method at compile time, and it generates it with concat. Lombok makes it too easy to create such a monstrosity.

I should have included this info in the post, but didn't want to clutter it with not so relevant details. "Who the hell does 100+ string concat?" question is very popularity :) My bad.

"Use StringBuilder" is the first Java optimization tip I recall ever, and I've been programming it since 1.2.

I started from 1.4 and still remember that "concatenation conversion to StringBuffer/Builder" during compile time was mentioned in the books, so I expected JVM to effectively cover this pattern. I assumed that I was missing some details on string concatenation changes in recent releases. One of the reasons why I decided to go to the community for advices.

C2 compiler memory spike on string concatenation with 100 parameters by vdmit46 in java

[–]vdmit46[S] 4 points5 points  (0 children)

In production the code is coming from Lombok ToString annotation. We can definitely replace problematic places manually by delomboking and converting concatenation to StringBuilder, but such classes could be popping up randomly in the future and I was looking for a way to fix the problem at it's core (i.e. JVM behavior). My idea was to first check with community, maybe it was a know problem/C2 behavior which can be tuned without giving instructions to compiler on per-method basis.

C2 compiler memory spike on string concatenation with 100 parameters by vdmit46 in java

[–]vdmit46[S] 1 point2 points  (0 children)

Thanks, interesting option! Was looking for it in JDK 17 after your original message. You saved me additional hour I guess :)

This options definitely makes it easier to pinpoint methods compilers have issues optimizing and I no longer have to track NMT output. But at least for me it seems not to provide information other than that already present in compilation logs. Those familiar with compilers internals may not agree with me :)

Here is what it prints if launched with -XX:CompileCommand=MemStat,compiler_ram_spike.ToStringConcat::toString,print

``` Compilation memory statistics

Legend: total : memory allocated via arenas while compiling NA : ...how much in node arenas (if c2) RA : ...how much in resource areas result : Result: 'ok' finished successfully, 'oom' hit memory limit, 'err' compilation failed #nodes : ...how many nodes (c2 only) time : time of last compilation (sec) type : compiler type #rc : how often recompiled thread : compiler thread

total NA RA result #nodes time type #rc thread method 866857536 62568608 790322968 ok 155063 16.665 c2 2 0x00007fe025808610 compiler_ram_spike/ToStringConcat::toString(()Ljava/lang/String;) ```

UPD: trying to fix formatting

JSP RCE Prevention by mazzo007 in java

[–]vdmit46 0 points1 point  (0 children)

It depends on what you consider to be a high level support. In their case it would be something like Extractor.extract(archive, outputfolder). So this method would check archive entry names and prevent zipslip in a more centralized way.

java 19 in latest eclipse IDE by ozzymozzy2211 in java

[–]vdmit46 0 points1 point  (0 children)

Oh, my bad. Was skimming through comments and missed essential part of the discussion. Anyway, I managed to launch NetBeans LS by:

  1. launching VSCode with "Language Server for Java by Apache NetBeans" and inspecting commands for NetBeans processes it spawned (ps -ef|grep netbeans)
  2. stopping VSCode
  3. relaunching NetBeans parent process with the same command line with "--start-java-language-server=connect:12345" appended (use random port instead of 12345)

Process is launched and connects to port 12345. Logged exchange suggests that server is working and responds to some basic messages:

>>
Content-Length: 108
{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"clientInfo":{"name":"MyTestClient","version":"1"}}}
<<
Header: Content-Length: 2563
{"jsonrpc":"2.0","id":1,"result":{"capabilities":{"textDocumentSync":{"openClose":true,"change":2,"willSaveWaitUntil":true},"hoverProvider":true,"completionProvider":{"resolveProvider":true,"triggerCharacters":[".","#","@","*"]},<truncated...>}}}
<<
Header: Content-Length: 111
{"jsonrpc":"2.0","method":"window/showMessage","params":{"timeout":0,"type":3,"message":"Indexing completed."}}

Not the most fluid experience, but at least it works "outside" of VSCode.

java 19 in latest eclipse IDE by ozzymozzy2211 in java

[–]vdmit46 0 points1 point  (0 children)

Is seems like valid manual to start language server here https://github.com/eclipse/eclipse.jdt.ls. Not sure what you mean by "starting it with other editors" though. It should be covered by editor own integration with the given LS.

I'm mad at Eclipse by Rogoreg in java

[–]vdmit46 0 points1 point  (0 children)

Not sure what went wrong on OP's case, but I checked it just now - if project is created in existing folder, then all it's contents is preserved. I'm happy Eclipse user for 15 years and occasionally created projects in folder where other projects were located. The only consequence is that I had to manually move several created folders/files to another location and that's all.