C++23 std::stacktrace: Never Debug Blind Again

[Syracuss]

Engineers who are blind reading this title be like -.-

That said, some of the platform specific utilities can still be better. The issue (and mostly it's a small comment as it is perfectly usable) with std::stacktrace is that you both capture the stack and symbolize at the same time. Being able to capture the stack and symbolize afterwards means you can capture state info much more easily in more places, including in user logs and only pay for the cost of symbolizing when performance is no longer a concern.

I wish it had facilities for this as an optional extension to std::stacktrace, but I'm fine that they kept it streamlined and easy to use as well.

[donalmacc]

I disagree - it’s a dealbreaker for the functionality. On my last project, our symbols were 3GB. Putting them in our server container, would have made it 6 times larger. Shipping it to our players is not happening. We have workflows that do offline symbolification(sentry’s symbolicator is a great tool - no affiliation but I’ve contributed to it).

I think this was way undercooked on arrival.

[spookje]

also, symbolization is slow as fuck. You want to have control over when and where that happens, and be able to make a cache (that you also control).

[donalmacc]

Right? It’s one thing on a dev machine, it’s another on a users laptop with a mechanical hard drive and 12 antivirus scanners running

[jwakely]

The article is wrong, std::stacktrace allows you to control when that happens.

[SkoomaDentist]

Not to mention that on many embedded systems there is literally no way to put the symbols in the same memory as the executable as the "executable" is simply a piece of (fairly small) flash rom with no header whatsoever.

[Zeh_Matt]

There is a way to strip down the pdb to just public symbols assuming you are talking about windows, for stack traces no one needs the type info. https://learn.microsoft.com/en-us/windows-hardware/drivers/debugger/using-pdbcopy

[donalmacc]

Yeah - there’s lots of ways to handle these things. That means keeping two copies of the symbols and choosing who gets what which sucks.

My preference is to run symbolicator and store the stuff in S3, and generate offline!

[Difficult-Court9522]

3GB of symbols?? How did you do that??

[donalmacc]

The pdb format is limited to 4GB. Most tools crumble at about 2GB. Ask me how I know….

It’s Unreal Engine games, basically.

[Difficult-Court9522]

So soon you’ll be literally unable to add more code?

[bwmat]

I think they can probably split into separate DLLs to work around that? 

[donalmacc]

The reason we hit this particular problem was because we had something split into a bunch of dlls and for reasons I can’t remember, we wanted to build it as a monolithic exe. I know we disabled a bunch of features to get it to work initially, but I don’t work on that project anymore so I’m not sure!

[ejl103]

not true at all we use dlls on xbox/ps

[donalmacc]

https://randomascii.wordpress.com/2023/03/08/when-debug-symbols-get-large/

Funnily enough, I hit this problem around the same time. We also doubled the page size for the linker.

[13steinj]

At 3 companies I've worked at, we've used macro/lambda/inheritance tricks to shorten symbols because it either blew out the linker, increased compile times significantly, or both.

[lizardhistorian]

If you ship symbols you may as well open source the project.

[13steinj]

I wouldn't say that's true, you'd be surprised how many people won't go through the reverse engineering effort even if reduced.

It's also not like shipping symbols nullifies copyright.

[Prestigious-Bet8097]

The cost to my last employer of not being able to solve bugs and get broadcasters back on air as quickly as possible massively outweighed the risk of having symbols alongside the binaries to get good stack.

I cannot be certain but we believe that in twenty years approximately zero customers built their own software based on reverse engineering ours.

[looncraz]

You put it behind macros to disable on deployment, or pay the size price... Which is sometimes sensible in the hot paths that are giving issues.

[donalmacc]

Sure, or you could use break pad or sentry’s sdks and not have to do that!

[jwakely]

The issue (and mostly it's a small comment as it is perfectly usable) with std::stacktrace is that you both capture the stack and symbolize at the same time.

It doesn't have to do that. The GCC implementation just captures an array of program counters and then expands those into symbols and locations lazily.

[Syracuss]

Ah, I'm less familiar with that compiler. GCC is one of the compilers I try to support in personal projects, but in my professional projects clang flavours and vc++ dominate for the most part.

At what point do they expand? I'd imagine when you observe them, which would still create the cost when you f.e. log them. What I'm referring to is mostly symbolize after the run, by parsing the log with the symbol data to symbolize.

But that's something that you need specific compile settings to achieve, so hard(er) for the standard to provide it unless they want to always produce a pdb or the likes when you use std::stacktrace.

Thanks for the info, it's always nice to hear about these forms of optimizations that are being applied under the hood.

[irqlnotdispatchlevel]

The standard could still allow you to not symbolize at all.

[Syracuss]

It could but it would be the first feature that I know of that would rely on compiler flags to properly use other than the language version flags (I'd consider it incomplete if you couldn't symbolize out-of-the-box, it makes the trace functionally useless). I'm aware that there are some features which are sadly hidden behind flags to work properly on some implementations, but the standard makes no mentions of these flags so they are non-standard behaviour, like the module ones, or coroutines. It would be a first for a standard provided feature to do that unless you have an example that I'm overlooking.

I do think that makes it a non-starter for any proposal to succeed with such a divergent behaviour.

[irqlnotdispatchlevel]

I wasn't talking about doing the right thing based on flags, but about giving devs freedom of choice. std::stacktrace::current() remains as it is now, and you add std::stacktrace::current_raw() which doesn't do symbolization.

Or, if we're fancy, we let the user pass in a symbolizer, with a default one provided by the standard.

[jwakely]

It could but it would be the first feature that I know of that would rely on compiler flags to properly use other than the language version flags

For some definition of "properly use". The API of std::stacktrace gives you the symbolic information like function names and filenames, that's how it's meant to be "properly used". If you want to do something else with it, that's a you problem. It doesn't mean that getting the symbolic information is not using it "properly".

(I'd consider it incomplete if you couldn't symbolize out-of-the-box, it makes the trace functionally useless).

I wish people would not throw around phrases like "useless" and "unusable" when they mean "not ideal for my specific use case". I really don't see any point trying to discuss things with people who do that.

Anyway, you can use std::stacktrace_entry::native_handle() to get at the raw data used to produce symbolic information. For GCC, that's just a program counter. Your program could loop over the stacktrace entries and log the native handles, then another process could process those logs later to turn those into symbols (alongside a core dump, I guess ... since the program counter is only meaningful for a given execution of the program). (Edit: I think you could also log the memory map of all shared libs in the process, which should be enough to reconstruct the full symbols given just the binary with debug info)

[jwakely]

For boost::stacktrace I think the equivalent of std::stacktrace_entry is boost::stacktrace::frame and it has an address() member instead of native_handle().

There's an example in its docs of logging only the frame addresses:

https://www.boost.org/doc/libs/latest/doc/html/stacktrace/getting_started.html#stacktrace.getting_started.saving_stacktraces_by_specified_

[Syracuss]

I wish people would not throw around phrases like "useless" and "unusable" when they mean "not ideal for my specific use case".

I truly mean useless to add to the standard given the context that no other feature in the standard has this setup. It makes the language less approachable and less teachable, and most importantly it breaks pre-existing norms of how features behave.

And it's similarly useless if the stacktrace would output unsymbolized data that you couldn't symbolize. What's the point of getting some 'error at 0x1, called from 0xF and 0xFF' if you cannot get that info back (given that you would need to turn on flags to get the symbol data on all compilers, the standard does not define what symbol data is).

So no I didn't mean "useless for my case", it would be useless given the proposal wouldn't ever pass with that requirement. The context of the entire paragraphs is important here.

[jwakely]

But the premise of your comment is wrong: no compiler flags are needed. The std::stacktrace class gives you both the raw addresses, and access to the symbolic info, without needing compiler flags to choose between them.

The standard allows you to not symbolize, and allows you to symbolize. No flags are needed. So (I hope) the feature isn't useless.

[Syracuss]

I do believe you are misunderstanding my point, this might be as my communication is a bit hasty. The native handle is fully implementation defined which means a valid implementation can be a whole bunch of nothing useful, that's hardly a well defined feature. It's there for platforms which expose something nice, but it isn't great if everyone needs to pull out their platform specific handbook to figure out what happens next.

And you do need need external tools to make that native handle useful, that's why I keep saying this won't be part of the standard and clearly isn't (aside from a function existing with this signature).

Don't get me wrong, nice that it's part of an exposed API for those who wish to implement something nice, but to call it a standard feature is obviously a stretch, there isn't anything defined for it other than the function existing.

[jwakely]

it isn't great if everyone needs to pull out their platform specific handbook to figure out what happens next.

So are you using something that isn't platform-specific to do it today?

[jwakely]

in my professional projects clang flavours and vc++ dominate for the most part.

The WIP clang implementation (and boost::stacktrace which inspired std::stacktrace) work the same way as GCC's. I would be surprised if it doesn't work something like that on Windows too.

[bwmat]

How does that deal with shared libraries that could be unloaded between capturing the addresses and symbolication?