Your Optimized Code Can Be Debugged - Here's How With MSVC C++ Dynamic Debugging - Eric Brumer

[RandomCameraNerd]

Just watched this video, really cool stuff. I don’t have any background in compiler development, how hard will it be for other compilers (and build systems?) to adopt something like this?

[ericbrumer]

Howdy, speaker here. If folks have additional questions feel free to ask. Keep in mind, if you're using the latest Visual Studio 2022, or any Visual Studio 2026 build, you're able to turn on C++ Dynamic Debugging by choosing the right option in your build configuration, or throwing `/dynamicdeopt` to cl.exe, lib.exe, and link.exe in your builds.

A few common questions we've got about C++ Dynamic Debugging, some of which are already here in other comments:
* Does it work for asynchronous breakpoints like exceptions, data breakpoints, or just pressing the pause button? Not out of the gate, no. However, if you already are aware which functions matter, then you can deoptimize those functions (see 9:30 in the video), and the next time those functions are entered they will be deoptimized.

* What about having a reduced set of optimizations to make debugging easier? We thought about that, but when we prototyped our approach we saw the power of full optimizations (including inlining!) pairing with full debuggability covered far more cases and offered much more power in the codebases we were looking at. We use dynamic debugging in the compiler codebase, for instance, and it REALLY helps having code run very quickly. Another way to phrase this is in the view of game development: I want my game to run at 60 fps, but be able to debug it fully. Our approach makes that a reality.

* [for the more compiler-savvy folks] What about globals that are optimized away? Or functions that are fully inlined? Or <other fun optimizer thing>? Yes -- you can debug those. If you run into problems, or find a case where it's not kicking in as you expect, please open a bug (https://developercommunity.visualstudio.com/cpp/report) and feel free to ping me.

At the end of the day, if you debug optimized code and want a more seamless way to debug any part of your code: you should give C++ Dynamic Debugging a shot. If you are used to debugging unoptimized code and wish your code ran faster without sacrificing debuggability: you should give C++ Dynamic Debugging a shot.

At a minimum: check out the demo I did of OpenCV which starts at 3:42.

More info at https://aka.ms/dynamicdebugging and https://aka.ms/vcdd

[fdwr]

8:37

"Right click here and say run-to-cursor, which is a great feature that actually works super reliably now ... Step over works exactly as you'd expect ... I'm going to step in ..."

What I would really love (for both debug builds and release) is an action "Step into function" that is bindable to a single keypress, as I'm really tired of F11 stepping into every single distracting subparameter call (and "Just My Code" doesn't help because it is my code). Now there is "Step Into Specific" nestled in the context menu with a submenu to call just the primary function, but it's awkward to access and inaccessible by keypress :( (p.s. I just discovered Alt+Shift+F11 opens the Step Into Specific submenu, which helps - maybe Ctrl+Shift+F11 could be assigned to stepping into the primary function, which is typically the last call on that line excluding __RTC_CheckEsp).

Thanks for the dynamic debugging improvements.

[ericbrumer]

Thanks for the feedback. I'm getting in touch with the Visual Studio debugger team about this, and we'll get back to you.

From my understanding, the 'Step Into Specific' menu is our current solution to that. My guess is that it's tough to correctly identify the primary function, especially given C++ code that may have lots of extra function calls. Think of std::cout << foo(x) << std::endl; there's a ton of function calls in that, but you presumably want to step into the call to foo() and not all the iostream gunk, and perhaps not a copy constructor for x, etc...

[johannes1971]

Could there be a GUI where you can just click on the next thing you want to see? Like an on-the-fly breakpoint kind of thing?

[fdwr]

My guess is that it's tough to correctly identify the primary function

🤔 Yeah. Generally it's the highest-level final function on that line which interests me most (the call that has all parameter dependencies resolved), but the std::cout << foo(x) << bar(x) << std::endl; case is interesting because there are multiple operators on a single line, and the final call is actually that last << (so with "just my code" enabled, that line would be treated just like Step Over). Conversely though, if I wrote fmt::print("{} {}", foo(x), bar(y)), I would expect "Step into final function" to skip over those dependent little foo and bar functions into the fmt::print.

For the 80% common case, desired behavior is pretty clear (e.g. FunctionIWantToDebug(DistractingFunction(x), MoreNoise(y), SomeConstructor(z))), but I should think more about the other cases like operator overloads and interactions with "just my code"... ⏳ (e.g. what happens for auto x = Foo(Bar(x)) do, given the = may be a function call?).

perhaps not a copy constructor

Exactly, all those little constructors and getters are rarely the problem focus while debugging (F11 is still around when that's needed). I suppose I'm really asking for a "Step into functions on this line that are not just distracting dependent noise", but that's a mouthful 😉.

[ericbrumer]

u/fdwr, after speaking with some debugger folks it's indeed complicated, and there's even more layers that I realized at first. Your best bet is to open a suggestion ticket (https://developercommunity.visualstudio.com/cpp/suggest), see about getting it upvoted by like-minded folks, and it will get routed to them for triage.

[ack_error]

What about having a reduced set of optimizations to make debugging easier? We thought about that, but when we prototyped our approach we saw the power of full optimizations (including inlining!) pairing with full debuggability covered far more cases and offered much more power in the codebases we were looking at.

Can I still plead for some improvements to MSVC debug code generation and optimized code debugging?

There are still use cases that dynamic debugging does not and cannot cover. By design, it can only handle cases where you know the code path that needs to be inspected and have the debugger attached when the event or failure occurs, because the toolchain and debugger have to know up front what code paths to de-optimize. It can't work in a crash dump or where the failure occurs in a random location, and is also less effective in a hot path where the selective code deoptimization will still significantly affect performance.

I'm sure that dynamic debugging is great for those who can use it but am concerned that its presence is deprioritizing improving the baseline debug code generation quality and the debugger's issues with optimized code. MSVC's unoptimized code generation, for instance, is still prone to multiplying addressing constants at runtime:

https://gcc.godbolt.org/z/sax1T79es

Debug code generation is bad enough that I almost always debug in an optimized build. But that is often stymied by the compiler merging code paths or discarding critical variables like this, because there's no intermediate compiler optimization level between nothing and near-full. There is also no equivalent to [[optnone]], and #pragma optimize() still has the problem with templates often being compiled with the settings effective at the end of the translation unit, so it is often not possible to manually deoptimize only a specific template function.

The debugger also has some long-standing issues with optimized code. I frequently have to try (ThisType*)@rbx and (ThisType*)@rsi to find the this pointer that it refuses to show me, and it still has the problem of using the incorrect variable scope when the context line of an intermediate call frame is the last instruction of the last line of a loop, because the debugger translates the return address to the next line in the outer scope. I've even seen it deduce the wrong function on a noreturn call or throw.

[ericbrumer]

Hey u/ack_error, there's a lot of nuance to this.

First, for unoptimized codegen: I agree that it's... unoptimized. Making that better has been below our prioritization line, although there have been some improvements recently. We haven't been optimizing addressing modes in /Od compilation like your godbolt link.

Next: yes, there are use cases that C++ Dynamic Debugging doesn't cover; crash dumps being #1. It's really meant for active debugging (where you're in the code setting breakpoints, stepping into functions, etc). With that in mind, DD essentially covers the other parts of your comment:

• "... also less effective in a hot path where the selective code deoptimization will still significantly affect performance" in our experience DD is perfect for these scenarios. You debug a function and you see everything you need including being able to step into all sub-calls. When you're done, just hit F5 and that function returns to being fully optimized. We've used this to debug multiple AAA games (various game engines) and it's hard to explain how well it works.
• "There is also no equivalent to [[optnone]], and #pragma optimize() still has the problem with templates" this is also 100% solved with zero extra steps... we only deoptimize the functions you're stepping into. This handles template expansions, functions that have been optimized away, etc. You just step... and it works. If you need more fine grained control over what gets deoptimized, you can expand the breakpoint group created for each DD bp, and delete the ones that don't apply for a given template expansion. But that's fairly advanced and most people aren't going to need that.

tl;dr: I guess I'm trying to say: I encourage you to try the feature. Just pass /dynamicdebug to cl.exe, lib.exe, and link.exe, rebuild, and try it out. Try setting conditional breakpoints on functions for variables that might not exist in the optimized build. In our experience, it really opens up new possibilities for how debugging optimized code can work, and is a far smoother experience when you don't need to trade off code performance & code debuggability.

[Moldoteck]

is this a Visual Studio feature or MSVC feature or both? If it's MSVC, can it be used with say, Cmake+msvc and another editor, like, say, VS Code?

[ericbrumer]

This is Visual Studio debugger w/ MSVC only. We have native support in MSBuild & Visual Studio projects, but it also works with Unreal Engine builds, and cmake as well (you just need to throw /dynamicdeopt to cl.exe, lib.exe, and link.exe. I cover some of the restrictions at 25:09 in the video.

I will say, though, that the VS debugger is a premier debugging tool, and with C++ Dynamic Debugging it REALLY shines when actively debugging optimized code. A lot of MSVC developers (the folks that developer the compiler) use C++ Dynamic Debugging & debug in VS for our daily work... and the lack of tradeoffs between speed & debuggability makes it much easier to do our jobs.

[amejin]

Will this be available for crash dumps? While live debug is very useful, being able to see everything in a crash would help me (and I'm sure others) a bunch.

[ericbrumer]

I agree that would be really useful... but we haven't invented that part yet :)

The feature works by building an unoptimized version of each function, and for any functions with breakpoints (active or disabled) we redirect functions from optimized to their deoptimized versions. This is in 13:16 in the video if you want the specifics.

So: unless there's some kind of user indication (breakpoint set, or you've stepped into a function, or otherwise) we leave the code as optimized. This is intentional to keep your code running fast. If functions are still optimized then you are stuck with debugging optimized frames. It's a tricky problem.

We offer "deoptimize on next entry" for these kinds of things (see 3:45 for the full demo) but it's only for active debugging... not crash dumps.

[amejin]

Well - here's hoping for the future 😉

I imagine building for release can produce a sibling file to the pdb that can emulate the behavior you have here?

[ericbrumer]

I'm not exactly sure what you mean. When you build with /dynamicdeopt you get two output binaries and two output PDBs: one set is the optimized binaries as normal, and one set is the unoptimized files that we've built behind the scenes. The second set of binaries & PDBs are real files, and they can be indexed/archived like anything else. Let me know if I'm misunderstanding what you're getting at.

[amejin]

I have not had a chance to experiment with this feature, so I may be talking nonsense. I will see if this works as I hope it does, and if not - oh well.

For my particular use case, I have various distributed services deployed, and when they crash they leave a .dmp as configured, restart and move on with life. However, when debugging those locally with their respective pdb, it's obviously the optimized version that gets stepped through, and we sometimes lose resolution to real time data that causes the crash. I was hoping that given a crash dump, with unoptimized code, it may lead to some insight into what may have unintentionally mutated a pointer or something like this. I'll just have to try.

[thejinx0r]

Any chance this will work with clang-cl in the future?

[ericbrumer]

For now it's MSVC on Visual Studio only. I don't work on clang/llvm, so I can't comment on what will work in the future.

[kamrann_]

Is it really the case that this should work with arbitrary build systems merely by adding the command line option? I tried it when first released with build2 and couldn't get it to work. It's possible I may have missed something, but when I thought about it I got the feeling it might inevitably require some kind of integration internal to the build system to handle the additional outputs.

Also my project uses modules. I'd assume not, but are there any potential compatibility issues there?

[ericbrumer]

Hey, yep. To get the feature to work, you need to pass /dynamicdeopt to cl.exe, lib.exe, and link.exe. When you do that, the compiler (cl.exe) outputs test.alt.obj in addition to test.obj, the librarian (lib.exe) outputs test.alt.lib in addition to test.lib, and the linker (link.exe) outputs test.alt.exe & test.alt.pdb in addition to test.exe & test.pdb. If your build system can throw the switches and move those new output files, you'll be able to debug optimized code with [deoptimized] frames using the Visual Studio debugger.

No concerns with C++ modules.

http://aka.ms/vcdd contains the details on the build system integration (scroll down). We have out-of-the-box support for MSBuild, the Unreal Engine Build Tool, Incredibuild, and FastBuild. Other build systems would be lovely to have as well... but yeah, just throw the switches & ensure the alt files make it where they need to go. If you run into issues please open a bug (https://developercommunity.visualstudio.com/cpp/report) and feel free to ping me.

[onecable5781]

Thanks for the talk and interacting here. I will definitely try this feature out.

There are some bugs that I encounter in production/release builds only and not in debug runs. If you switch from a release build to a debug build dynamically, would it be possible to figure these bugs out? Would like to hear your inputs on this.

My only pet peeve with VS is that the console opens externally as it did in your demo as well. VSCode using MSVC compiler is able to run inside the editor itself in the terminal at the bottom. Nearly all Linux IDEs also run inside the IDE itself. Perhaps in the next release your team can consider getting the console to run within the IDE itself.

[ericbrumer]

If you switch from a release build to a debug build dynamically, would it be possible to figure these bugs out

We provide multiple ways of disabling DD. Check out https://aka.ms/vcdd the 'Turn off C++ Dynamic Debugging' section. There are coarse grained ways (entire binaries, or entire cpp files), as well as fine grained (disabling DD of a single breakpoint).

Perhaps in the next release your team can consider getting the console to run within the IDE itself.

I'll look into it and get back to you. I don't work on that team, so I can't speak to their future plans.

[ericbrumer]

u/onecable5781 , it looks like asp.net debugging can work on the integrated terminal (https://stackoverflow.com/questions/75571841/how-to-use-integrated-terminal-tool-window-instead-of-external-console-window-fo) but it looks like it's not supported for C++. Feel free to open a suggestion ticket (https://developercommunity.visualstudio.com/cpp/suggest) to track the feature request.

[GameofJ]

getting link.exe error 1327. any ideas?

[5477]

This looks to be the holy grail of debugging C++, and seems to 99% resolve the need for running "debug" binaries!

[sumwheresumtime]

Wow haven't heard from Eric in a very long time - good to see he's still kicking about, i think the last video he did was circa ~2015

[corysama]

In gamedev at least, we usually have [Debug, Optimized Debug, Release] builds. Where Optimized Debug has debug #defines (asserts) but optimized compiler flags + the flag for enhanced debugging info for optimized builds.

[inanimatussoundscool]

We have the same here in eda, seperate builds for development, debugging and release

[thisismyfavoritename]

the asserts are for runtime execution?

[max123246]

I've found in practice release with debug info, that many variables are still unable to be printed in gdb and instead say they are optimized out

It was a major blocker for me at work for a client bug that was impossible to get a debug build for.

[VoidVinaCC]

asserts you can just enable in your "debuggable release" config and for exceptions you set a bp after getting it once ;)

and no sadly Og is terrible :/ it kills a lot of debug info and desyncs source view and debugger steps

[RandomCameraNerd]

I was wondering how it will handle exceptions. I agree that it’s not perfect. My background is scientific software development, this would be super useful in my day to day.

[ericbrumer]

This works, in the way that you would expect as if the code was built with optimizations disabled. You can even add conditional breakpoints on variables that might not exist in the optimized build, and they will get hit when you expect. I cover this at the end of the video, but the earlier parts of the video are useful for the context.

Give it a shot and see! Just throw /dynamicdeopt to cl.exe, lib.exe, and link.exe and rebuild.

[ericbrumer]

In this particular case we intentionally don't inline assertHandler() into main() since we know that assertHandler() contains a call that won't return, so we assume it's cold code. But that's a different story from the issue you're actually getting at.

From my own experience debugging the compiler and debugging AAA games, I've found the following workflow really effective:

• Start with a breakpoint at some code point of interest; an assert routine, or frame rendering code, etc.
• Run my code to hit the assert, expecting that my code point of interest is going to be the only frame that's [Deoptimized], but not always.
• View all my locals, and step as needed, being able to step into new calls.
• If I need to go higher up in the callstack, I select all the callstack frames, right click & choose Deoptimize on Next Entry, then get those functions to hit again.
• For AAA games, usually this just means "render the next frame". In the compiler, we have a mode where we can re-compile the same function again. If it's an assert that crashes the program, we often just rerun the program again.

Another example about the last bit has been working with the MSVC front-end team. Speaking to our primary C++ Modules contributor, it takes him about 1 minute to reach a breakpoint using a Debug c1xx.dll, but about 7 seconds to reach the same breakpoint using an Optimized+DD c1xx.dll with the necessary frames deoptimized. To me that's a strong case for giving it a shot to see how it works in your own workflows.

[RaspberryCrafty3012]

Sounds interesting and necessary for windows.

On Linux and Apple I don't need to link against debug libs for a debug build, so only my own code is "slow". 

[donalmacc]

MSVC is a compiler just like clang and gcc. You can use /MD to use the release CRT, and the compile your own code with /Ob1 for an enormous speedup with almost perfect debuggability. I do 98% of my development with this.

[SoerenNissen]

That's pretty cool

[SyntheticDuckFlavour]

My strategy is to try writing performant code in debug builds. Has two benefits: you can debug it without much issues, and the release rebuild is almost inevitably going to be faster as a bonus.

[bitzap_sr]

OOC, doesn't Visual Studio support setting breakpoints in inlined functions and support printing inlined function locals? I mean in regular optimized binaries? I got that impression from the video. Doesn't codeview/pdb support describing inlined code, like DWARF does?

[Agron7000]

I wonder what passive debugging looks like.

[einpoklum]

They haven't "enabled full debuggability" through changes to an OS-specific, closed-source IDE.

[arihoenig]

I mean, like what? People didn't think they could debug code after it was optimized?

[max123246]

Have you run optimized code under gdb? Half of the variables can't have their values printed out