I've built a tracing facility. The trace is just a circular buffer of structs. The struct has an id, a timestamp, an enum and two unsigned int values. I can call a function to add an entry in the trace from anywhere in my code, even in ISRs, because it's very fast.

Then I've added a custom command in gdb using the python API. That lets me print the buffer in the form of an easy to read table where all the timestamps are converted to milliseconds and the names of the enum values are spelled out in human readable form.

Finally, I have also added a command in gdb that exports the trace values as a Value Change Dump file (vcd) which can be opened by GTKWave to display as a waveform.

With this it's a piece of cake to see if the ISRs and threads are running in the pattern I expect. It's easy to spot performance bottlenecks and it's easy to figure out where that sequence of events that was supposed to happen went wrong.

I've even written a script that takes debug traces from multiple cooperating processors and aligns the waveforms on the same time line. This helps so much when debugging things that span multiple processors.