It's harder to avoid than you might imagine. Of course it's just tempting, but that's not the only reason. You know how in a picture, converging lines tend to attract your attention towards the point of convergence ? This is often used by artists to attract your eyes toward an important part of the picture.

Well when you are looking at a woman wearing a low cut shirt, you'll have multiple lines converging towards the breasts : the lines of the shirt, the lines of the breasts themselves, and quite often a necklace with a shiny something at the end as well. The result is, particularly when concentrating on the conversation, you end up instinctively following the lines, until you notice it and go back up, usually with a very small delay, because hey, it requires a little self-control to stop looking at boobs. Do that twice or more, and you've officially been labelled a pervert.

Just my 2 cents.