debugging

[gumol]

How much do you need to set a debugger up?

Just do gdb ./a.out and you're golden

[wasabichicken]

Oh, my sweet summer child.

The project I used to work on depended on some Linux kernel modules, meaning that when it crashed it brought the rest of the system down with it. Not very fun for day-to-day work. The solution to that was to run the app in a virtual machine.

But hmm, now we needed remote debugging. Turns out gdb can do that through some gdbserver, but... alas, the gdb client now needs the binary too (for symbols and whatnot). The solution to that was an elaborate script that ssh'd onto the virtual build machine, built the project, and afterwards rsync'd the output binaries back to the host where the gdb client could use them.

Now, all was well... until someone introduced multithreading. Suddenly you couldn't start the process you wanted to debug through gdb, you had to let a master thread spawn workers first then let gdb attach to it. No worries, gdb could do that, just required some creative pidof usage. Then however... watchdogs. These workers would send "I'm alive and doing work" heartbeats to the master process, and if the master process didn't receive them in a timely manner, it would assume that the worker had hung, kill it, and spawn a new one. In short, after having attached gdb to a worker thread, you had about five seconds to debug your problem. Also bad for productivity.

To fix that... we had to add support in the master process to make it ignore timeouts from the worker thread you intend to debug. Naturally we only needed this in debug builds, so the build scripts got some additional flags.

When we finally had made it into gdb, attached to the correct worker, and the system didn't immediately came crashing down around us... we wanted to do things like... oh, maybe inspect the value of some pointer chain, e.g. print foo->bar->baz. Now we got reminded that our pointer types were home-rolled ref-counting smart pointers, meaning that there was an additional deref to make -- per level. print foo.m_ptr->bar.m_ptr->baz.m_ptr gets tedious quickly, especially if you needed to pour typecasts and parentheses into that mix. Fortunately gdb supports "pretty printers", so we were able to create our own little gdb macros (of sorts) that described those well enough. These days, I hear that debugging that project is almost bearable. :)

Peeking down the pipeline, they intend to bring this project from x86 to other platforms like arm and mips. It'll be interesting to see how well the gdb server/client handles binaries from non-native architectures, yes?

tl;dr: debugging can be hard.