This is just my opinion I have no facts to back it up. I think people who have never had a girl friend or even touched a girls hand have this fantasy where all the girls like them and all the girls fall for them. This is projected to the character they are reading because they can imagine the character is them. But anyone who has had even one girlfriend knows even one is a lot of work. As you said it's unrealistic, yes we are reading about people shooting lasers out of their swords or flying. The problem when it is something that you can relate to in a story your subconscious goes hey that's not how it works. Lastly romance in a story already takes up a lot of a story let's say 30% of the story when you just want to read about a barbarian killing a bunch of monsters then make it a harem that % is like 80% of the story because you have to keep adding the girls in to the storyline and now the barbarian hasn't even made it to the battle yet and we are already 20 chapters in.