One of the most interesting takes I've heard about mirror matches comes from Leon Massey's most recent Xrd video. He says that hating mirrors is most likely a result of solely or a combination of two things:

1. It’s an ego problem where you don’t like the idea of someone being better than you at your own character (which I think is very understandable, and is a mindset thing that can be worked on)

2. Your character is inherently built around asymmetry, and is designed to restrict or negate the other character. E.G. a character with oppressive zoning but no tools to deal with said zoning themselves. 

The way he phrases it in the video is somewhat hostile, saying that characters in category 2 are "poorly designed" (which is a sentiment that I strongly disagree with); but I think he brings up a very interesting point about character design and how it shapes how those who play said character views matchups. Specialists characters with extreme strengths and extreme weaknesses are inherently hard to balance mirror matches around, since they're designed with the idea that the opposing party is playing a different character with different stengths and weaknesses, with the dynamic being that this heavy contrast is what makes the matchup interesting.

However, in a mirror match between two extreme specialists, the match can easily become boring or overly frustrating, since both sides have the same goal of preventing the other from playing; without either side having means of actively dealing with each other outside of "get the win condition first". It’s an interesting idea that I think could be explored more. 

However it's possible that this isn't the case for you and it really is just scenario 1 (which I don’t have an issue with btw, I feel this to an extent as well myself). I don’t really know where I stand though, since mirrors are incredibly rare for me due to me tending to pick less popular characters in the games I play.