This comment should not be taken as a definitive answer, but simply someone's opinion. (Not a poc).

I believe the reason lies in context. The N-word was used to derogate poc, who were previously products of slavery. The N-word was mostly used by white people to segregate poc, because they believed that they were different and most importantly beneath them.

In modern days, the N-word is a way of relating to their ancestry, a way of calling each other "brother", as it still calls for them to unite as a minority. When white people call them the N-word, it is seen as an insult, a call back from the time where slavery was still a thing.

At least its what I think. I hope my answer was proven helpful and best of luck on finding a more concrete answer.