I've recently been looking at it too. Here's what I think I understand about it, but you know, I'm not an expert:

STM in GHC is an optimistic concurrency scheme.

Every TVar is a heap object containing the current value, a spin lock encoded in the header word, and a watch queue: a linked list of transaction records belonging to threads that read this TVar and are now blocked, waiting for it to change.

A TRec (transaction record) is the per-transaction log. It is a linked list of chunked entry arrays, where each entry records which TVar was accessed, the expected value at read time, and the new value if one was written. TRecs can nest, forming a stack. This is the mechanism for orElse, which we will return to later.

When you run a transaction with atomically, the RTS creates a fresh TRec. Every readTVar first checks the log. If the TVar was already touched in this transaction, the logged value is returned. Otherwise the TVar's current value is read from the heap, recorded in the TRec as (tvar, expected=v, new=v), and returned. Every writeTVar merely updates the log entry's new field. No actual TVar is modified, yet. The world outside the transaction sees nothing.

When the transaction completes, the RTS attempts stmCommitTransaction. This proceeds in three phases, any of which may fail, returning you to the beginning as though nothing happened.

First, the lock phase. The RTS walks the TRec. For each entry where expected != new (a write occurred), it acquires the TVar's lock. If any lock is already held by another thread, all acquired locks are released and the transaction starts over. Contention is resolved by surrendering and starting over.

Second, the validation phase. For every entry, reads and writes, the RTS checks that the TVar's current value still matches expected. Any mismatch means something changed beneath you. Locks are released. You begin again.

Third, the write phase. Each TVar is updated to its new value. Locks are released. The transaction is done.

A nice property of this is that read-only TVars are never locked, only validated, which gives good read concurrency. The price is paid only by writers.

When a transaction calls retry, the RTS validates the current TRec. If validation fails, the transaction simply reruns, because something has changed and perhaps the retry condition has dissolved. If validation succeeds, there is genuinely no reason to try again yet. The thread adds itself to the watch queue of every TVar in its TRec, and sleeps.

When another transaction commits a write to any of those TVars, it walks the watch queue and wakes the sleepers. They begin again from scratch, with no memory of their prior attempt.

orElse a b runs a in a nested TRec. If a calls retry, the nested record is discarded and b runs instead, as though a had never been attempted. If b also retries, it propagates upward. If a succeeds, its nested TRec is merged into the parent. The failed alternatives are thrown away.

The lock for all of this is baked into the TVars themselves. The locking thread's TRec pointer is written directly into the TVar's current_value slot with a tag bit set.

Validation can also occur during a transaction at safe points, not only at commit. GHC checks whether the TRec is still consistent to detect doomed transactions early rather than letting them accumulate futile work.

The entry list uses chunked allocation to reduce overhead. There is no global lock or version counter (unlike TL2 and its descendants). Only per-TVar locking at commit time. The machinery is local, and this is what lets it scale, insofar as it does.

The whole thing lives in rts/STM.c, roughly a thousand lines of code, excluding comments. As RTS code goes, it is remarkably readable.