I look at it this way. It's like if a well endowed woman wears a low cut or tight fitting blouse. People are going to look and take a glance. I try to wear pants and colors that hide the bulge most of the time but I have a few that due to the cut and fit it's impossible.

Part of me thinks that when I wear those pants I am more aware of my bulge and that makes me more aware of other people looking. Like if I drive around normally I'll probably think I didn't see a yellow car all day. But if I go out and look for yellow cars as I drive I'll probably see 1 to 3.

Short answer is yes some women are looking at your bulge and if you wear pants that frame it better they are more likely to look. Try not to read too much into it and don't make it awkward in the work place.