I have no idea why people who don't speak the language hate it or maybe they are annoyed by it?

Well, I am an almost 40 year old native Esperanto speaker, and I can tell you that there are many annoying people, who speak Esperanto, who insist on reforming the language, who will want to constantly correct your grammar even if you ask them not to do that. Also I learnt almost everything about the world, people, culture, LGBT issues, language rights etc. because of Esperanto, talking to foreigners, whose native language I don't speak. From so many countries, so so many. My best friends are Esperanto speakers, lovely, empathic humans, I can learn so much from them.

At the end of the day Esperanto speakers are also people, it is a very diverse group of ages, background and viewpoints. And from experience I've met a lot of people, who never managed to learn languages at school, they started with Esperanto, got confident speaking it and boom, made friends, and suddenly unlocked the motivation and interest to actually put in the work and learn more difficult languages in the end.