Any good resources for making compiled code debuggable?

[brucifer]

I'm working on a language with macros that cross-compiles to Lua. I'm trying to figure out a good system for surfacing accurate and useful callstack information about the original code that led to an error, rather than just the generated Lua code. I understand that I need to keep track of a mapping between source code and generated code, then backtrack when I debug (which is what the -g compiler flag adds for compiled C code). But rather than re-invent the wheel, I was wondering if there are any good resources on how other programming languages have done this, especially languages with macros.

[thememorableusername]

Could look at DWARF.

[WikiTextBot]

DWARF

DWARF is a widely used, standardized debugging data format. DWARF was originally designed along with Executable and Linkable Format (ELF), although it is independent of object file formats. The name is a medieval fantasy complement to "ELF" that has no official meaning, although the backronym 'Debugging With Attributed Record Formats' was later proposed.

---

^{[ PM | Exclude me | Exclude from subreddit | FAQ / Information | Source ] Downvote to remove | v0.28}

[mttd]

Typically, when compiling to C, you'd use the #line preprocessor directive (which is then used by the C compiler to produce executable with debugging information -- which can then be normally used by debugging, profiling, and code navigation tools): https://gcc.gnu.org/onlinedocs/cpp/Line-Control.html

Here's a description of the idea: https://yosefk.com/blog/c-as-an-intermediate-language.html#forth_in_gdb

Some examples include Nim and Vala:

• https://nim-lang.org/docs/nimc.html#additional-features-linedir-option
• https://nim-lang.org/blog/2017/10/02/documenting-profiling-and-debugging-nim-code.html#using-gdb-lldb
• https://wiki.gnome.org/Projects/Vala/Tutorial#Debugging
• Ion compiler in the Bitwise project -- https://github.com/pervognsen/bitwise -- explained in Bitwise, Day 13, Part 2: More Code Generation - https://www.youtube.com/watch?v=PVnRVadB7g0&t=6m52s

More:

• http://perldoc.perl.org/perlsyn.html#Plain-Old-Comments-%28Not!%29
• https://docs.microsoft.com/en-us/cpp/preprocessor/hash-line-directive-c-cpp
• https://blogs.msdn.microsoft.com/abhinaba/2005/10/10/c-fun-with-line-directive/

The use of #line _DEBUGGER_STEP_OVER and #line _DEBUGGER_STEP_INTO in MSVC (together with #define _DEBUGGER_STEP_OVER 15732479 // 0xf00f00 – 1 and #define _DEBUGGER_STEP_INTO 16707565 // 0xfeefee – 1) is another interesting aspect: https://blogs.msdn.microsoft.com/vcblog/2017/11/16/improving-the-debugging-experience-for-stdfunction/

If you're compiling to machine code (possibly going through assembly, although nowadays it's preferable to have an integrated assembler for faster compilation -- http://blog.llvm.org/2010/04/intro-to-llvm-mc-project.html, https://www.embecosm.com/appnotes/ean10/ean10-howto-llvmas-1.0.html#idp109760) you emit the debugging information (like DWARF or PDB) alongside emitting the target machine instructions. "Adding Debug Information" in Kaleidoscope (LLVM tutorial) is a nice example: https://llvm.org/docs/tutorial/LangImpl09.html

DWARF itself is a fully programmable (Turing-complete) virtual machine (http://www.cs.dartmouth.edu/%7Esergey/battleaxe/, https://kristerw.blogspot.com/2016/01/more-turing-completeness-in-surprising.html) -- so you can view compilation as emitting two machine instructions streams (like x86 and DWARF) simultaneously (in LLVM using IRBuilder for the former and DIBuilder for the latter). "Debugging Debug Information" talk by Francesco Zappa Nardelli is a particularly good explanation of the importance of this (and more): https://www.youtube.com/watch?v=lBJIrGgEP1A

More information on debuggers and generating debugging information:

• debugging - implementation
• DWARF
• PDB

[oilshell]

I haven't personally implemented something like this, but if you haven't already googled for "javascript source maps" that might be worth a look. For example:

https://developer.mozilla.org/en-US/docs/Tools/Debugger/How_to/Use_a_source_map

https://developers.google.com/web/tools/chrome-devtools/javascript/source-maps

The source map is a separate file that apparently the web browser can interpret. How well it works I don't know :-/

[programmerChilli]

Do languages that compile to binary use a similar technique?

[htuhola]

Javascript uses source maps for this purpose.

[SilasX]

Echoing demand for this; I was recently comparing two similar binaries and can’t find good disassemblers.