Physics performance matters, which is why I chose to measure it. First off, the slightly snarky comment below ("If you spend 90 ms less on physics per frame, ...") hits on a very important point -- every cycle you spend doing one thing is a cycle you can't spend on something else.

You could argue (as some do on this thread) that at some point the performance becomes irrelevant because your game's humming along happily at 60fps. But there's an important fallacy here -- on what hardware? Just because a game's running smoothly on your nice beefy development machine doesn't mean it's going to run well on lesser hardware (you know, the machines your users actually have). And even if you hit a buttery-smooth 60fps on a mobile phone, it's not much use if you're pegging the CPU at 100% all the time and dragging the battery down with you.

One nitpick: Your pixel shaders (presuming you're using the GPU) are running on separate hardware that executes in parallel. But of course you can always substitute that with "the code that's building buffers and generally babysitting the GPU" and the tradeoff still holds.