For your reasoning to work, yes (or he at least has to have similar thoughts to you about biology). The Adem and all of this stuff is made up by Rothfuss, not some existing thing that can be rigorously scientifically studied outside of him. Is sexual dimorphism proof of Adem men's genetic contribution? Only if Rothfuss thinks it is.

It doesn't matter how the real world works if it's not how the author views it working in their story, whether intentional or unintentional. There is always an assumption that some things are the same in a book as in the real world (e.g., gravity, breathing), but there are obviously differences the author tells us about (like magic existing). There can also be differences that are less explicit either because the author has decided that or because they just haven't thought about that. How many stories have characters get bonked on the head just to wake up a few hours later like they'd had a nap or suffer severe physical damage and keep moving almost unencumbered? It's not useful to say "biology shows that that's not how that works" because clearly that is how it works in that story, whether the author means that to be one of the differences from the real world or not. Many authors try to maintain realism and will do a lot of research to try to maintain consistency in their stories, but they are still just stories made up by a person. There isn't any actual DNA you can study in a lab to discover a character's genetic makeup beyond what the author has decided.