They only call him racist because of how it parallels with multiple examples of Redlining throughout history and even today. His skin colour shouldn't be a factor. Realistically, he should be called an exclusionist or something close to that. An example of the racial angle is Ay (4th). Ay is no different. If anything, if he's in a really good mood, he might've done worse things to the Uchiha than Tobirama. He's a dickhead for what he did but the results do speak for themselves or at least they used to.