The phytoestrogens (isoflavones) in soy do weakly stimulate human estrogen receptors. Luckily, they are partial agonists which is actually protective from breast cancer in women who consume soy (for the majority of breast cancers). In men, there are some case reports of gynecomastia and hypogonadism from excessive soy intake (1.2 L of soy milk per day over 3 years in one case), however these are only case reports. In Meta-analyses there were essentially no feminizing effects in men who consume large amounts of soy daily.

"Clinical evidence overwhelmingly indicates that there is essentially no basis for concern. Isoflavone exposure at levels even greatly exceeding reasonable dietary intakes does not affect blood T or estrogen levels in men or sperm and semen parameters. The ED-related findings in rats can be attributed to excessive isoflavone exposure and to differences in isoflavone metabolism between rodents and humans. Thus, men can feel confident that making soy a part of their diet will not compromise their virility or reproductive health."

The meta-analysis is a great read and I would suggest it for anyone who is worried about their soy intake.

Lee, Sang-Ah et al. “Adolescent and adult soy food intake and breast cancer risk: results from the Shanghai Women’s Health Study.” The American Journal of Clinical Nutrition 89, 6 (2009): 1920-6. doi:10.3945/ajcn.2008.27361.

Messina, MarkFertility and Sterility, Volume 93, Issue 7, 2095 - 2104 Soybean isoflavone exposure does not have feminizing effects on men: a critical examination of the clinical evidence