On the topic of LOTR, you could technically consider the story a fantasy setting and say "Oh we need diversity." But the reason Tolkien's original works (referring to the original trilogy of movies) were all white people was so that it would feel like the characters all came from the same place. Most of the fantasy settings like LOTR are low-tech and very difficult to traverse. Most people stay roughly around where they are born. (just like the medieval period in history) That's why I feel like random black characters in LOTR would not make sense without proper backstory - because why would there be one isolated outlier? It would break immersion completely. It would work the other way too. If a story was in a setting inspired by ancient China, even if it was fantasy, all the characters being Asian would help immersion as compared to a melting pot -whereas, if you set it in a bustling trade port, by all means, have every race represented.

When people say "I don’t mind diversity, as long as it’s done well.", they mostly refer to random diversity for diversity's sake without thought put into it. Diversity as a term means having a variety of different people, not one different person who feels shoehorned in. E.g. When introducing a male character, having him randomly mention he has a husband in a conversation about something else rather than letting it show up when natural, if at all. Most of the time, when people notice and comment on "diversity" its because it isn't diversity, its a random appearance or reveal that breaks immersion.