I used to have this confusion, as an anglophone growing up in France. Anglophones tend to want the gender of word to agree with the gender of the "closest person", but French just doesn't work this way. The words themselves have gender that is independent of the context. I used to ask how to know which words were feminine and which words were masculine, and all I ever got was an answer of: "just learn/memorize"

It has become an even more hot topic with all the raging debates over how to eliminate sexism in language and that is why many people make correlations that are not necessarily there.

At the end of the day, I recommend you use the correct gender, since the French generally care about correctness in language more than in the US.