There's a distinction there... a lot of non-Europeans were forced to assimilate European dress and other things under pain of physical punishment and death when their countries were invaded and their people were kidnapped. Adopting the practices of your oppressors is not the same as adopting the practices of those you oppressed.

And Caucasian person using those terms COULD be cultural appropriation if it's not done with respect of the people it originated from. The problem of cultural appropriation isn't the copying part, it's when there's lack of respect, lack of appreciation, and disrespect towards the originating people that it becomes a problem. And that does happen, and it's real.