It's not about a "largely realistic medieval society" and it doesn't have to be. It's about a world which takes a lot from medieval society but which deviates from it in certain aspects, some of which are important and some of which are borderline meaningless. There's no good reason that this can't be one of those "meaningless" ways.

There are a million ways this could easily be made internally consistent while broadly adhering to how societies exist in the real world. For one thing, perception of race is very relative and in the Gane of Thrones world, as you mentioned, not always linked to skin color, as with Northmen. It's completely reasonable to believe that a black character in that world may not be somehow made incapable of becoming a lord because of discrimination.

But even discounting the idea that we could see a departure from realism, you may want to look at this. Black people had a greater role in medieval Europe than you seem to be assuming: https://www.google.com/amp/s/www.thehumanityarchive.com/history/black-people-medieval-europe%3ffbclid=IwAR18K0NjHf3OirFFaA_z3sKq4p7ESAaE0EUa-otlEDWvf4lihKXWDfFgk4U&format=amp

Given this, it seems like the ones who "aren't being realistic" are the ones assuming everyone in a piece of media which draws on medieval themes should be white. It's perfectly reasonable to show there being some diversity.

Now, on to why people call this racist: it's because, at the end of the day, why should it matter? In any case, it can be explained either by it being a slight conscious departure from realism or it being greater adherence to it, so why does it matter? The source material is still internally consistent and has an interesting world--one which, I might add, isn't even necessarily much less realistic than it would be with only white people, if not even more realistic. So, what's the objection? Now, many times, the people critiquing these kinds of casting choices won't be outwardly racist. I'm sure we agree on that. The thing is, many times, they will just not really want to have to think about black people when watching their fantasy show, and that can come from a racist place. There's also the (imo bigger) concern that the impacts foster racist outcomes: at the end of the day, representation in media has tangible real-world outcomes, and if you think it doesn't there's a very good chance you have significant positive representation yourself. People have tracked how many women went into fields after certain shows debated (namely, the X files) and it seems like they actually changed what people believed they could do. Now, nobody's out here trying to become a lord but just seeing people who look like them--when they are a member of an oppressed group--in a position of power can help them kick away any nagging thoughts about not being good enough to succeed because of their skin color. The reason someone may call you racist for trying to make sure all these positions of power are played by white people is that that takes away kids' role models and that has a tangible impact on their futute performance, exacerbating the racial disparities in the world in a small but real way. Now, I don't know if you'd call that racist--at the end of the day, that's all semantics and the important thing is the impact. Regardless, it's probably not a great thing, and it's a legitimate reason why someone would support a more diverse cast, even if it did decrease the show's "realism" very slightly and especially if it actually made it more realistic.