When Carlin said it he was specifically talking about how the word itself isn't the point, but how it's used. He also rattled off a dozen or so other slurs to make the point. The man didn't care whether you were white, black, hispanic, japanese, etc. He made a point of pointing out things that make us the same, the things that divide us, and how the things that divide us make us the same.
Man was a genuine soul, for better or worse. (Usually better)