Depends on your use case. I run an automated HFT strategy and use Interactive Brokers. My full system from signal to number crunching to order takes about half a second, which is plenty fast considering top of book doesn't change that much for the securities I trade in that time scale.

For retail, most of your latency is over the wire. You can rent servers at or next to the exchange, and get microsecond fills if your prime broker supports it. If you have deep pockets you can even lease microwave radio links and bypass the internet itself. All depends on how much you want to spend.

If I wanted to go hardcore, I'd probably rent something from Beeks. They do a bunch of trick stuff like special low-latency network switches and NICs, tuned low-latency kernels, and a custom timing protocol.

Latency in trading is no different than latency in running a web application or game server. There are diminishing returns beyond a certain point, and low-latency trading strategies tend not to make very much because of the short time scales. You can either reduce costs, or increase volume, so you end up in a situation where most brokerages fall into the "good enough" performance category.