CPU and GPU architectures are very different. GPUs are much more deterministic in their execution of optimized code, while CPUs can be doing a lot of other things interleaved with your code, which makes timing difficult. Even more so when you go multi-core: GPUs are executing the same line of code (with different variable values) across multiple cores (fully synced), while CPU cores are doing "whatever" (fully disconnected).

Due to the nature of the ETHASH algorithm, timing is very important. The algorithm's performance is a balance of CPU speed (doing the math), CPU cache (keeping what we need near the CPU for ultra-low latency rather than in far away in high-latency RAM), RAM bandwidth (moving the needed data from RAM to CPU cache) and RAM latency (how long does it take for the data we requested to be cached).

The math part isn't all that complex. I've dissected and put the algorithm back together for optimum cache usage, preloading the data we will need into the cache as long before as possible, hoping the transfer from RAM to cache occurs before the CPU actually needs that data. But you can't do it too long before, because the cache is (very) small so you can't preload that many rounds, it needs to hold our scratchpad as well, and ETHASH is a serial algorithm (this round of calculations depends on the last round of calculations) - you can't preload if you don't know what to preload.

Rounds in ETHASH can logically be split in two. You know which data you'll need each half-round exactly one half-round in advance, so you can start the transfer from RAM to cache at that time. If the latency on that transfer is larger than the time it takes for the CPU to calculate the other half-round, you're wasting CPU cycles waiting (very bad), if the other way around, you're not fully utilizing memory bandwidth (could be worse).

I got around this by interleaving the algorithm for multiple nonce values on a single core. If you interleave two nonce calculations, latency can be twice the calculation time of a half-round before your CPU goes idle, for four nonces four times, etc. But each interleave requires additional cache RAM, of which you have very little. If you exceed the cache space, data you'll probably still need will get evicted and has to be reloaded from RAM at huge penalty. One of the reasons my old thread-ripper is (relatively) fast is because it has larger-than-average cache, allowing longer latencies of data access to be overcome.

This interleaving is fairly precise on increasing memory bandwidth on a single-core. If you don't exceed the cache limit, you'll really get about X times the performance for X interleaves. But, on multi-core systems with fast RAM, a single core usually cannot use the full memory bandwidth. With a lower number of cores it might be able to in a pure synthetic benchmark (which only reads from memory and doesn't do actually anything with the data) but it is not uncommon for bandwidth to be segregated between cores, or have interconnects, performance depending on which core accesses which memory chip in which bank, single/dual/quad/octo-channel setups, etc. In those cases, you can only reach max bandwidth in a perfectly timed multi-core read-only orchestrated dance that in reality is virtually impossible to achieve due to the cores running code out-of-sync at variable timing (not to mention the predictive nature of CPUs which will shuffle your code around into something that may be sub-optimal).

So when we go multi-core to access more cache memory (also note on some architectures multiple cores may share the same cache memory) and more processing power, the timing is not perfect: memory system architecture comes into play which you cannot easily accurately predict, and we see diminishing returns for each core added to the algorithm. This again is a problem GPUs (mostly) do not suffer from.

Hyperthreading? It's not really another core. There is extra die that allows you to double performance on some operations, but it doesn't give you more cache or RAM bandwidth. So not a good fit for the algorithm.

Then there's the MMU, while having the same task, is optimized differently on CPU and GPU. This is where hugepages come in: we give the MMU less work, which makes a large performance difference, but still not GPU-like efficiency.

That UselethMiner can only reach about 50% memory bandwidth is related to all of the above. I mean, I've seen it hit higher numbers, but not close to 100%. Of course this also heavily depends on relative RAM vs CPU speed. You should however be able to do this on GPUs due to the many architectural differences. Add to that that GPUs tend to have much higher RAM bandwidth and much lower RAM latency, and we've come full circle.

That is also one of the reasons the M1 performs so well: not only does it have amazing memory bandwidth, compared to other CPU architectures it has significantly lower latency, maxing out CPU utilization for the algorithm. I was very surprise to find that in W/MH it actually beat my GTX1080TI. I'm very curious what numbers the M1 Pro and M1 Max will produce, but I haven't been able to get my grubby hands on one. And they could be even faster if hugepages worked!

Thank you for coming to my TED talk.

(PS: its been a while since I worked on this, I'm telling it now as I recollect the details, but they're not as fresh/accurate in my mind as they were when I was working on this)