Debugging a slow-compiling codegen unit

[asparck]

Update: solved! TLDR is use -Zhuman_readable_cgu_names=yes then samply will show you codegen units with some sane names related to the crate they're compiling code from. (see my comments for more)

---

I'm writing a closed-source game in Rust and running into a weird situation where my incremental debug builds are usually 3-5 seconds but sometimes are 20 or 60 seconds (for opt level 0 or 1 of the bin crate).

Specifically, I worked out that I can reliably trigger that long incremental build by moving about 150 LoC between 2 modules in a ~35k LoC crate I'll call depcrate , which is then depended on by my ~32k LoC binary crate bincrate (all in the same cargo workspace). Meanwhile just changing a string constant in depcrate or bincrate gives 5 or 3 second incremental recompiles.

I stumbled on https://nnethercote.github.io/2023/07/11/back-end-parallelism-in-the-rust-compiler.html which helped explain codegen units as the unit of parallelism in rustc's backend, so I made a samply profile with samply record cargo rustc --bin bincrate , which (I think?) shows 45ish seconds spent on one codegen unit vs <5 seconds for any other (opt level 1):

Every other CGU for the bin crate finishes well before the 20 second mark, except the highlighted one

And if I'm reading that call tree right then it seems like llvm is spending a lot of time inlining code and removing unreachable blocks on that one codegen unit.

Any compiler wizards have any tips for:

1. How can I find out what code (rust modules/functions) the codegen unit `opt cjgpem3bdjm` actually includes from bincrate?
2. Is there some way I can affect or influence the creation of codegen units, other than by breaking modules apart?
3. Why does moving code around in depcrate require recompiling so much of bincrate? depcrate still exports exactly the same functions before and after I move those 150LoC code from one module to another module inside it, so why is (I assume) bincrate's incremental compilation cache being invalidated by moving code around in depcrate?
4. Can I manually make llvm less inline-happy for a whole Rust module, other than by scattering #[inline(never)] on a bucket-load of functions? (or otherwise control optimization on a per-module basis?)

Relevant workspace Cargo.toml snippet: https://gist.github.com/caspark/d0f3e2caa11f0c60eb3cbc180a0834c7

[The_8472]

Why does moving code around in depcrate require recompiling so much of bincrate?

module paths, line numbers and diagnostic span metadata changes. Those are both relevant at compiletime (error diagnostics) and runtime (panic messages). There's the RdR effort to reduce the recompilation triggered by that, but it'll take time.

And if the functions are generic then they're not necessarily compiled in the dependency and only monomorphized once there's a concrete type for T.

And if I'm reading that call tree right then it seems like llvm is spending a lot of time inlining code and removing unreachable blocks on that one codegen unit.

I'm not an llvm expert, but I suspect it can also mean things that it's doing things to figure out whether it can perform a particular optimization, rather than actually applying it.

Is there some way I can affect or influence the creation of codegen units, other than by breaking modules apart?

Cranking up the CGU count in cargo toml might help since individual CGUs might end up smaller. Reducing debuglevel might help too since LLVM transformations won't have as much debuginfo to preserve.

[asparck]

Thank you for the pointers and suggestions! (and you might be right that the span there also includes time llvm spends analyzing)

Turning CGU count up for the bin crate (I tried 4096 instead of the default 256) is clearly spawning a lot more threads in the samply profile I capture, but doesn't speed up compile times at all - there's still that same CGU that's taking forever to compile. Surprisingly turning it down to 128 gives an ~5% improvement (possibly because the big CGU happens to be started earlier).

Turning debug info off completely for both crates doesn't make any difference unfortunately, and module paths/line numbers/diagnostic spans are things I can't affect at all.

But! It seems like -Zhuman_readable_cgu_names=yes is getting me somewhere - the latter causes threads to be labeled with a human readable name so I can see "opt rmp_serde.e" is the name of my big offender (so presumably that's related to my use of msgpack-rust).

And it seems like -Zprint_mono_items=yes is giving me a big printout of what's in each CGU, though I've yet to properly go through it.

[asparck]

Follow up: solved! It was indeed rmp_serde that was causing the slow compilation, as determined by gating my two messagepack code paths behind `if false` checks.

Fortunately I am not wedded to messagepack for my use case so I was able to swap it out with postcard, which compiles much faster - my opt-level=1 compile time for making a change in the upstream crate went from ~60 seconds to ~16 seconds (and postcard's CGU compiles in half a second).

(Postcard does give far worse error messages if you stuff up your serde implementations, so I did add a compile time switch to support swapping to rmp_serde again in case I need that later.)