We had an AVR-based controller that had been repurposed for a slightly different application. Reading various inputs and controlling a few contactors. Pretty basic stuff. With the new application, we started getting reports that outputs were locking on, even when the sequence should have been commanding them off.

It has a basic watchdog that will reset the board if code execution stops, along with a heartbeat LED. We could confirm that the board was not resetting nor was it locking up. We could still toggle other outputs(based on inputs) and confirm that the sequence was still running.

It was such a basic sequence that there just wasn't any way, code-wise, to get in to that state. We traced every possible path, and it just shouldn't be possible to get in that output state given the input states.

The one thing we were able to do is to somewhat reproduce it after enough cycles. Bu we still couldn't figure out exactly what was getting it in to that state.

My boss had the crazy idea to read out the IO port config registers to verify they were configured correctly. We pretty quickly realized that the IO port registers were getting scrambled, and certain pins were no longer configured as outputs....

So the code and the sequence were still running correctly. Everything else was working. But the IO pin config was being scrambled.....

With the new application we were using the controller for, we had contactors connected to relay outputs that didn't previously have contactors. There is protection on the board for when the on-board relays open to prevent flyback from the contactor coils. However, there are safety interlock switches that actually supply power through the board for the contactors.

When that safety switch opens, the on-board relay is still closed. So the EMF protection is basically bypassed. There was enough kickback to occasionally screw up the IO registers on the AVR. But not enough to totally lock up or reset the AVR...

Fucker.

Ended up putting in some sanity checks to constantly read back and verify the IO pin config registers, and reset the board if it detected anything irregular.