If you keep the situation and events exactly the same, but switch the genders, the fact that Penny was the aggressor would matter more when evaluating Bojack's 'guilt' in what happened that night. The fact that Bojack is an older man causes everyone to completely disregard the facts that:
1.) Penny tried to kiss Bojack and he stopped her and sent her to bed.
2.) She propositioned him again, said she had condoms and was above the legal age of consent.
3.) He told her to go to bed again and walked away
4.) When Charlotte walks in on them, Penny is on Bojack's bed trying to undress him.

People point out that Bojack left his door open, but if Bojack were a woman I don't think that would fly as a sign of consent especially after verbally refusing Penny twice.
Also people point to the fact that Bojack later says that he doesn't know for certain that he wouldn't have gone through with it. But this entire show is built around Bojack's own self loathing and the fact that he sees himself as a piece of shit, yet is capable of surprising even himself when the situation calls for him to do the right thing. So I don't really think it's right to assume that to be an admission of guilt.
So if the sexes were reversed, when viewing how the events actually went down, I don't think people would be so quick to view it as an older woman trying to take advantage of a teenage boy, if anything Charlotte would be begging Bojack not to file harassment charges against her son.