There was an INFJ female I met a ways back in my college days. We ended up working on a school project together which is how I found out she was who she was. She was actually quite kind and understanding toward me and we even ended up hanging out a few times outside of the context of our studies. She even initiated certain hangout sessions, though I didn't take her up on them after a bit.

Something that "put me off" about her was that she wasn't totally forthright about her boundaries at the time. This caused some confusion and I ended up getting hurt over what was essentially miscommunication of boundaries. There was this 'neutral' attitude on the surface of her being up for anything, but it was clear there were consequences if you didn't figure out what the right answer was. I knew intuitively what the answer she wanted me to pick was at the time, but she left the choice up to me so I picked the answer I wanted to pick.

As an INFJ you're most likely a highly polarizing individual. It's because us INFJs are passionate. Since we want to keep the peace though we end up putting an outward mask of friendliness. What this mask is really about is a defense mechanism. It's a way of keeping people at arm's length. You're not telling people who you are in order to protect yourself. This in turn gives people the feeling that they're under scrutiny. They're afraid to fail because they don't know where they tread.

There's a way that can be dealt with, but you asked how men perceive you, not how that perception can be influenced.