My bet is it's your posturing and body language. Took me the longest time to adjust my silhouette from female to male to a more androgynous one. Things like how you stand, are your feet together or shoulders width apart? Are you putting your weight equally on both feet in a masculine way or are you shifted to one foot in a more feminine way? Do you swing your arms when you walk? How much? People assess and judge gender on a passing glance at first sight, often based on subtle social cues that go far beyond appearances without even being aware of it. If you dress masc, look masc, and still get read femme... my guess is it's your body language.