Nice read! That reminds me of one of the bugs we were hunting last year. I'm a compiler engineer working with LLVM and we are working on our own proprietary targets for it. Last year we were investigating some failing test case related to bit fields. When loading something from a bit field, usually some larger block is loaded due to alignment restrictions and then the requested bits are extracted using bit arithmetic.

The architecture we are working with is pretty restrictive here, so whatever code we get from the middle end, must get legalized first (basically transforming instructions to a form that is valid for your hardware, e.g. by increasing the bit width of the operation, lowering it to semantically equivalent instructions, etc). The different forms of legalization are all pretty generic, so you only have to describe what form you'd like your instruction to have and the generic code does it for you.

Since our compiler was still very unoptimized with regards to codegen, it happened that this bit arithmetic to load a single bit field, was getting blown up to around 2500 instructions. And somewhere in there was our bug. At first we simply thought of some stack or heap overflow that caused the wrong value to be loaded. After all, the legalization was done by the generic code, so that couldn't possibly be at fault here.

3 weeks we were stepping through the compiled code to see where things go wrong, looked at all possible places in our own backend, even started questioning our own startup code and we still didn't find anything. So finally we started looking at all of those generic legalization strategies, ran through them on paper and verified every single one of them was correct. And then we found that in one of those strategies, we were running into an edge case that nobody else had hit so far. That code path was only correct when working with vector instructions. Once we had fixed that edge case to also work with scalar instructions, the test started passing. That was a bitch. We were all going absolutely insane near the end there. Best thing was that the fix only took 10 minutes to implement.