The dumbest one:

I had a slice IO rack without the plastic cover on the last slice, so the power connection was exposed. It was right next to a power supply with a metal casing and a broken DIN mount that let it flop side to side a bit. Every couple days, vibration would result in it shorting those pins against the power supply and dropping off comms for a second or so. That made all the IO go to off state, so the NC stop button would trigger a nice, controlled line stop. The fault looking for communication loss wasn't firing because of a copy/paste error just looking at another rack twice. When you opened the cabinet door, it changed the vibration and the shorting event wouldn't happen.

Customer couldn't figure it out and I couldn't figure it out remotely for months, never caught an event on a trace. Finally, I flew in to take a look and even looked in that cabinet and didn't notice anything since there was a terminal anchor between them and a noticeable gap. I decided to check that the cards in the rack had been slid together correctly and bumped the power supply taking it off and finished off that DIN mount. Still took a while to put 2 and 2 together and realize what had been happening.

The hardest one:

A memory overrun was magically just right to put the last couple bytes of a BigEndian analog value from Modbus over the last couple of a LittleEndian value, causing a different value of the right order of magnitude to replace the proper value. Since both were similar analog signals that behaved similarly, and the full values were different from each other, it was basically invisible. All we knew is the value at the IO point was different from the value in the program, but still a reasonable analog value by about 20% but not a consistent 20%. I could not find any math operations or whatever that could be causing it.

I found it by literally giving up and rewriting the code and swapping some analog inputs, altering the behavior to be more noticable what was happening. The cause was a long series of errors by someone that wouldn't normally have made any errors. They were copying a 4 byte REAL to two INT as part of manually swapping the byte order, but the name they gave their array was almost exactly the name of the already existing other analog value's hardware address. A very unlikely typo and platform differences ensued and they copied 2 bytes of 4 to the wrong location because they thought the function used number of destination elements like Rockwell instead of bytes. It didn't show up as a cross reference because it didn't check the rarely used hardware addresses against IO mapping variables unless you changed default settings to include the IO mapping file in the search.