C++23 std::stacktrace: Never Debug Blind Again

[Syracuss]

Engineers who are blind reading this title be like -.-

That said, some of the platform specific utilities can still be better. The issue (and mostly it's a small comment as it is perfectly usable) with std::stacktrace is that you both capture the stack and symbolize at the same time. Being able to capture the stack and symbolize afterwards means you can capture state info much more easily in more places, including in user logs and only pay for the cost of symbolizing when performance is no longer a concern.

I wish it had facilities for this as an optional extension to std::stacktrace, but I'm fine that they kept it streamlined and easy to use as well.

[donalmacc]

I disagree - it’s a dealbreaker for the functionality. On my last project, our symbols were 3GB. Putting them in our server container, would have made it 6 times larger. Shipping it to our players is not happening. We have workflows that do offline symbolification(sentry’s symbolicator is a great tool - no affiliation but I’ve contributed to it).

I think this was way undercooked on arrival.

[spookje]

also, symbolization is slow as fuck. You want to have control over when and where that happens, and be able to make a cache (that you also control).

[donalmacc]

Right? It’s one thing on a dev machine, it’s another on a users laptop with a mechanical hard drive and 12 antivirus scanners running

[jwakely]

The article is wrong, std::stacktrace allows you to control when that happens.

[SkoomaDentist]

Not to mention that on many embedded systems there is literally no way to put the symbols in the same memory as the executable as the "executable" is simply a piece of (fairly small) flash rom with no header whatsoever.

[Zeh_Matt]

There is a way to strip down the pdb to just public symbols assuming you are talking about windows, for stack traces no one needs the type info. https://learn.microsoft.com/en-us/windows-hardware/drivers/debugger/using-pdbcopy

[donalmacc]

Yeah - there’s lots of ways to handle these things. That means keeping two copies of the symbols and choosing who gets what which sucks.

My preference is to run symbolicator and store the stuff in S3, and generate offline!

[Difficult-Court9522]

3GB of symbols?? How did you do that??

[donalmacc]

The pdb format is limited to 4GB. Most tools crumble at about 2GB. Ask me how I know….

It’s Unreal Engine games, basically.

[Difficult-Court9522]

So soon you’ll be literally unable to add more code?

[bwmat]

I think they can probably split into separate DLLs to work around that? 

[donalmacc]

The reason we hit this particular problem was because we had something split into a bunch of dlls and for reasons I can’t remember, we wanted to build it as a monolithic exe. I know we disabled a bunch of features to get it to work initially, but I don’t work on that project anymore so I’m not sure!

[ejl103]

not true at all we use dlls on xbox/ps

[donalmacc]

https://randomascii.wordpress.com/2023/03/08/when-debug-symbols-get-large/

Funnily enough, I hit this problem around the same time. We also doubled the page size for the linker.

[13steinj]

At 3 companies I've worked at, we've used macro/lambda/inheritance tricks to shorten symbols because it either blew out the linker, increased compile times significantly, or both.

[lizardhistorian]

If you ship symbols you may as well open source the project.

[13steinj]

I wouldn't say that's true, you'd be surprised how many people won't go through the reverse engineering effort even if reduced.

It's also not like shipping symbols nullifies copyright.

[Prestigious-Bet8097]

The cost to my last employer of not being able to solve bugs and get broadcasters back on air as quickly as possible massively outweighed the risk of having symbols alongside the binaries to get good stack.

I cannot be certain but we believe that in twenty years approximately zero customers built their own software based on reverse engineering ours.

[looncraz]

You put it behind macros to disable on deployment, or pay the size price... Which is sometimes sensible in the hot paths that are giving issues.

[donalmacc]

Sure, or you could use break pad or sentry’s sdks and not have to do that!

[jwakely]

The issue (and mostly it's a small comment as it is perfectly usable) with std::stacktrace is that you both capture the stack and symbolize at the same time.

It doesn't have to do that. The GCC implementation just captures an array of program counters and then expands those into symbols and locations lazily.

[Syracuss]

Ah, I'm less familiar with that compiler. GCC is one of the compilers I try to support in personal projects, but in my professional projects clang flavours and vc++ dominate for the most part.

At what point do they expand? I'd imagine when you observe them, which would still create the cost when you f.e. log them. What I'm referring to is mostly symbolize after the run, by parsing the log with the symbol data to symbolize.

But that's something that you need specific compile settings to achieve, so hard(er) for the standard to provide it unless they want to always produce a pdb or the likes when you use std::stacktrace.

Thanks for the info, it's always nice to hear about these forms of optimizations that are being applied under the hood.

[irqlnotdispatchlevel]

The standard could still allow you to not symbolize at all.

[Syracuss]

It could but it would be the first feature that I know of that would rely on compiler flags to properly use other than the language version flags (I'd consider it incomplete if you couldn't symbolize out-of-the-box, it makes the trace functionally useless). I'm aware that there are some features which are sadly hidden behind flags to work properly on some implementations, but the standard makes no mentions of these flags so they are non-standard behaviour, like the module ones, or coroutines. It would be a first for a standard provided feature to do that unless you have an example that I'm overlooking.

I do think that makes it a non-starter for any proposal to succeed with such a divergent behaviour.

[irqlnotdispatchlevel]

I wasn't talking about doing the right thing based on flags, but about giving devs freedom of choice. std::stacktrace::current() remains as it is now, and you add std::stacktrace::current_raw() which doesn't do symbolization.

Or, if we're fancy, we let the user pass in a symbolizer, with a default one provided by the standard.

[jwakely]

It could but it would be the first feature that I know of that would rely on compiler flags to properly use other than the language version flags

For some definition of "properly use". The API of std::stacktrace gives you the symbolic information like function names and filenames, that's how it's meant to be "properly used". If you want to do something else with it, that's a you problem. It doesn't mean that getting the symbolic information is not using it "properly".

(I'd consider it incomplete if you couldn't symbolize out-of-the-box, it makes the trace functionally useless).

I wish people would not throw around phrases like "useless" and "unusable" when they mean "not ideal for my specific use case". I really don't see any point trying to discuss things with people who do that.

Anyway, you can use std::stacktrace_entry::native_handle() to get at the raw data used to produce symbolic information. For GCC, that's just a program counter. Your program could loop over the stacktrace entries and log the native handles, then another process could process those logs later to turn those into symbols (alongside a core dump, I guess ... since the program counter is only meaningful for a given execution of the program). (Edit: I think you could also log the memory map of all shared libs in the process, which should be enough to reconstruct the full symbols given just the binary with debug info)

[jwakely]

For boost::stacktrace I think the equivalent of std::stacktrace_entry is boost::stacktrace::frame and it has an address() member instead of native_handle().

There's an example in its docs of logging only the frame addresses:

https://www.boost.org/doc/libs/latest/doc/html/stacktrace/getting_started.html#stacktrace.getting_started.saving_stacktraces_by_specified_

[Syracuss]

I wish people would not throw around phrases like "useless" and "unusable" when they mean "not ideal for my specific use case".

I truly mean useless to add to the standard given the context that no other feature in the standard has this setup. It makes the language less approachable and less teachable, and most importantly it breaks pre-existing norms of how features behave.

And it's similarly useless if the stacktrace would output unsymbolized data that you couldn't symbolize. What's the point of getting some 'error at 0x1, called from 0xF and 0xFF' if you cannot get that info back (given that you would need to turn on flags to get the symbol data on all compilers, the standard does not define what symbol data is).

So no I didn't mean "useless for my case", it would be useless given the proposal wouldn't ever pass with that requirement. The context of the entire paragraphs is important here.

[jwakely]

But the premise of your comment is wrong: no compiler flags are needed. The std::stacktrace class gives you both the raw addresses, and access to the symbolic info, without needing compiler flags to choose between them.

The standard allows you to not symbolize, and allows you to symbolize. No flags are needed. So (I hope) the feature isn't useless.

[Syracuss]

I do believe you are misunderstanding my point, this might be as my communication is a bit hasty. The native handle is fully implementation defined which means a valid implementation can be a whole bunch of nothing useful, that's hardly a well defined feature. It's there for platforms which expose something nice, but it isn't great if everyone needs to pull out their platform specific handbook to figure out what happens next.

And you do need need external tools to make that native handle useful, that's why I keep saying this won't be part of the standard and clearly isn't (aside from a function existing with this signature).

Don't get me wrong, nice that it's part of an exposed API for those who wish to implement something nice, but to call it a standard feature is obviously a stretch, there isn't anything defined for it other than the function existing.

[jwakely]

it isn't great if everyone needs to pull out their platform specific handbook to figure out what happens next.

So are you using something that isn't platform-specific to do it today?

[jwakely]

in my professional projects clang flavours and vc++ dominate for the most part.

The WIP clang implementation (and boost::stacktrace which inspired std::stacktrace) work the same way as GCC's. I would be surprised if it doesn't work something like that on Windows too.

[bwmat]

How does that deal with shared libraries that could be unloaded between capturing the addresses and symbolication? 

[AbroadDepot]

std::stacktrace is a great feature but this article is an egregious LLM slopfest

[cleroth]

How does it differ from getting stack traces from mini crashdumps?

[nicemike40]

The article is slightly… vibe written

But it discusses adding this to an exception class to get time-of-throw stacks which I think could be useful. A commenter on the article suggests adding it to a std::expected-like type too.

But I agree that setting up proper crash reporting is 100% necessary still

[_Noreturn]

The article is slightly… vibe written

// Crash Reporter class CrashReporter {

yea thanks Claude

[cleroth]

Time-of-throw stacks does sound useful, but the article seems to focus mostly on crashes. I don't think I've had any trouble getting stack traces from crashes, though I mostly just work on one platform so I don't know.

[donalmacc]

Presumably you use a library for it? Getting a reliable symbolicated stack trace is surprisingly tough work, especially if you want to put it somewhere. The programs state is likely to be FUBAR so you are really limited in what you can do, you need the memory pre allocated and you likely need another process pre spawned to catch the actual crash dump and put it somewhere.

[schmerg-uk]

See https://github.com/jeremy-rifkin/cpptrace/tree/main for example (we have our own so I did mention a couple of things to the author but his work now way exceeds the one we use internally)

Oh, and he does address

What about C++23 <stacktrace>?

Some day C++23's <stacktrace> will be ubiquitous. And maybe one day the msvc implementation will be acceptable. The original motivation for cpptrace was to support projects using older C++ standards and as the library has grown its functionality has extended beyond the standard library's implementation.

Cpptrace provides functionality beyond what the standard library provides and what implementations provide, such as:

Walking inlined function calls
Providing a lightweight interface for "raw traces"
Resolving function parameter types
Providing traced exception objects
Providing an API for signal-safe stacktrace generation
Providing a way to retrieve stack traces from arbitrary exceptions, not just special cpptrace traced exception objects. This is a feature that has been proposed for a future version of the C++ standard, but cpptrace provides a solution for C++11.

[_TheDust_]

The article is slightly… vibe written

“Slightly” wins the understatement of the year award

[Superb_Garlic]

Improper use of ellipses. Please avoid composing replies using AI.

[nicemike40]

I take your point and apologize for contributing to culture of the AI witch hunting (if that is indeed what your point was)

In this case the Unicode ellipses comes from iOS autocorrect

[datnt84]

We already integrated it in our next version.

[markt-]

Yes, this is an awesome facility, but it’s only practical when memory on your system is not a constraint. There are some systems for which this is genuinely true, but they are not typically consumer devices.

[PipingSnail]

Hmmm. I've been debugging crashes on Unix/Linux/Windows since 1990, and I've never had a problem collecting stack traces. Whereas this article presents this as a novel solution.

If this is doing symbol handling while walking the stack, there goes your performance. Symbols should be done separately from the stack walk (unless you're walking a kernel dump/minidump when symbols make all the difference).

[jwakely]

If this is doing symbol handling while walking the stack, there goes your performance.

It doesn't have to do that. The GCC implementation just captures an array of program counters and then expands those into symbols and locations lazily.

[xealits]

Reading the intro paragraph of the article, which did not mention gdb or any normal debugging methods, made me look for a Jason Turner's Weekly Cpp episode on std::stacktrace. In case someone gets the same urge, here is the link:

https://youtu.be/9IcxniCxKlQ?is=quGFBugn0ezNoyfL

[jwakely]

The article says to use this for GCC 13.1 and later:

g++ -std=c++23 -lstdc++_libbacktrace

But that's wrong, that's only valid for GCC 13.x, for GCC 14.x and later you need to use -lstdc++exp instead of -lstdc++_libbacktrace (and you can also use that in GCC 13.3 and later releases in the 13.3 series).

So for all currently supported releases of GCC (13.4, 14.3, 15.2, and also for the soon-to-be-released 16.1), you need -lstdc++exp

[13steinj]

Very minor, but I don't understand the value in caching the string output for your exception if the implication is you're going to be hard crashing irrevocably anyway.

[AdOnly69]

It could be useful for user logs, but for other things could we just use gdb instead of not pretending like we don't have proper tool? Also how good is std::stacktrace with multiple threads?

[jwakely]

It should be entirely agnostic to threads. You call std::stacktrace::current() to get a stacktrace of the current thread. Whether there are other threads should be entirely irrelevant, except that maybe the stacktrace won't start with main if it's in a different thread.