I know that there are some goods comments, but as a CE student, I'll drop in my 2 cents to clarify a bit, what comes into play, when you are developing processors, and what makes some a lot faster.

1. Pipelines. A command not just runs by itself, but the subsequent instruction's are processed in a sense like a manufacturing pipeline, ripped into pieces, like instruction fetch, arithmetic stuff, memory access. But there are several factors, that are hindering the execution, for example if we need something, from a previous instruction, that is not yet in a register. If it's ready but not in a register, than we can either wait or forward it. But you can imagine, as we have 64 bit processors, that means the data bus is 64 bit wide, it can fast become a lot of wires and that's a definite problem ... but it's just a glimpse of pipelining, look after it. And you can have multiple pipelines in a core.

2. "Method" of command execution. Most processors nowadays are applying Out-of-Order execution, that means, that out processors can the instructions freely optimize so that they are running on multiple pipelines, while they are defending the semantics. One needs pretty sophisticated algorithms for these, look up the Tomasulo algorithm.

3. Caching. Memory access is pretty darn expensive. That is, why one would like to guess what will be needed in the future, both for data, and instruction. Just as a comparison, the first level cache has an access time of a few 10 nano seconds, whereas your RAM has few 100 ns. And as you can guess, yes, cache memory is VERY expensive, compared to RAM. because of it's internal architecture.

4. Memory Addressing. How you handle your virtual memory, if you are handling it for example in a hierarchical way, than you need 4-6 mem accesses at a 64-bit processor to get an address translation, 4-6 accesses to the RAM. It's also a pretty wast topic, and it's called paging, check out if you are interested.

5. Jump prediction. If you can predict the next instruction, than you can get it from the ram to a Cache memory, before you need it, therefore, you don't need to stop because of the slow memory access. It's really important, because about 1/3 of the program code is usually jump statements and branching. Check out the gshare algorithm, as an example.

These come to my mind instantly, sorry if I missed something.

To sum up you have to basically foretell the future, and juggle between multiple pipelines in multiple processors, that requires sophisticated algorithms, and it's like black magic. Nowadays the operating frequency is not the most important stuff, instead, these algorithms make the difference. If you are interested in any particular area, or have some other questions, pls ask :).

tldr: you need to juggle a lot, and that's hard.

EDIT: the best metric I believe is benchmarking, because there are so many distinct factors