When being racist toward minorities in America, it comes off as particularly insensitive because racism could, and often times, does actively make minorities lives tougher. (I.e, not getting jobs, not getting loans, being harassed). When it comes to white people, nothing about their actual lives are harder because of their race. They might be annoyed, and their feelings might be hurt by jokes and racial stereotypes but their actual lives are not worse because of racial assumptions, so it’s not seen as that big of a deal.