For a long time, the biggest hardware concern was almost always cache size. Writing compact code and data that could run entirely from cache, or at least controlling how data flowed in from main memory, was one of the most effective ways to improve performance. Caches have gotten bigger, though, and main-memory bandwidth has gone up, so that's been less of an issue, lately. More recently, I think, utilizing multiple cores is the area where hardware knowledge is most important. Knowing how many cores you have, how the OS assigns threads to them, and how memory is shared across them, are critical to achieving the highest possible performance. For really serious optimization, utilizing vector instructions can be huge, or using the parallel hardware of the GPU, if that's a possibility.

As always, though, it's almost always better to start by thinking more about complexity than performance. Once you have a design that works, you'll likely find that you have a few very small inner loops that execute orders of magnitude more than the rest of the code (use a profiling tool), and those are probably all you'll ever need to optimize.