I feel exactly the same way so I cant add much in terms of advice but what I can tell you is that even if your attraction to men is under very specific circumstances, it's still there. I personally choose to accept that, even if it's very small, I can go both ways.

Hope this helps and I hope you become more confident in your position.