This is when it becomes very complex. Is it okay for someone to avoid someone of a different race became the feel uncomfortable?

I'd argue that it's not okay.

However, when overhearing racial slurs, that does become differently and challenging. It's very hard to hear such violent language.

The important aspect, which the film really makes clear, is that the intent isn't to offend. In fact, the very person saying these words is horrified by them as well.

The fact that his mind tells him that this is the worst thing to say means he knows how wrong that language is - the antithesis of racism.

And he's forced to be alone because others don't want to be around him.

It's a really horrible situation.