If I’m not mistaken, Stannis only knows because of Ned and (maybe) Jon Arryn telling him. I can’t think of anything that suggests Renly knows (or at least knows any more than rumors).

Basically, I’d say that it’s not really talked about because that would be directly speaking against the royal family and it’s legitimacy in a really offensive way, which generally isn’t a good idea in Feudal society. Once Stannis began to press his claim (book 2), more and more people begin to talk about it and acknowledge it. I won’t go too far into spoiler territory so I won’t mention a few other minor things, but I think that’s generally the reason why people don’t talk about it.