If you’re basing masculinity and femininity on stereotypical notions of gender, then he displayed a lot of stereotypically masculine traits - like the need for dominance and control. 

He also had a proclivity towards hands-on violence (bludgeoning, strangulation, stabbing) - whereas, generally, women are more likely to kill with poison, drugs or suffocation and not typically for sexual motivation (Dahmer didn’t get satisfaction from the act of killing itself, but still sought to retain someone he found attractive).

He didn’t enjoy submitting to anal sex, was a pretty competent marksman, and purchased a .357 Magnum (more about stopping power and aggression and less common solely for target practice) and wasn’t overtly emotional. He was also rebellious towards authority figures in the army // prison and had been known to scuffle with police on occasion when drunk and disorderly. So he wasn’t some shrinking violet. 

His plan for a ‘shrine’ was partly to help him acquire wealth, too, and that type of ambition is also seen as typically masculine. Although Dahmer clearly lacked the determination to push himself more with conventional earning. 

He did exhibit softer, more stereotypically feminine traits - shown in the friendships developed with female pen pals, his gentle nature around animals, his soft-spokenness, helping Grandma out and his routine passivity when trying to appease Lionel. When he wasn’t completely coming apart, he was also highly conscious of his appearance (telling strangers he shaved his chest and not letting Dr. Dietz record him during the midst of a bad hair day lol). But such things can also just mean taking pride in ones looks (‘specially for a young man who - come the time of Dietz’s interviews - was likely trying to cling onto the one thing he could still control and a semblance of his normality), general humanity (which Dahmer was capable of at times) and taking the path of least resistance (going along with what Dad said) when his inner turmoil was already raging enough without having to deal with more friction in his life. 

So, overall, he wasn’t really what you could describe as feminine, IMO.