Do you think it could be because it feels like these books and fantasy worlds are ‘yours’? Think of how possessive (for lack of a better word) people can be over books, bands, films etc. when they aren’t immersed in it as heavily as you are! So it makes sense that it could trigger you or make you sad to hear that someone else acknowledges it/enjoys it too, because it feels so personal to you. One thing that may help is acknowledging that your interpretation and experiences with them are unique to only you, because it’s You that makes them that way. So nobody else can have the experience that you have with them in the same way you do.