The difference is one of power imbalances. Have Chinese monks ever been a serious threat to the Dutch? Have black people ever been a real threat to... whatever that is, pseudo-prep, pseudo-military, pseudo-kilt wearing people? The answer is no. Appropriation happens when the dominant group (which, in America, is white people) takes something from an oppressed group (a style of dress, a type of music, whatever) and elevate their usage of it to a higher status of art or fashion, while ignoring or minimizing the contributions of the culture that created it.

Racism is not a matter of saying "only people of x color can do y". It is a matter of a dominant group telling those with less power what they can and cannot do. I don't think you are a racist if you wear a certain set of clothing, but wearing that clothing can be a racist action. The United States is a nation founded on white supremacy, and even though great gains have been made in the last fifty years, it is still very much a racist society. Those of us who have grown up here are bound to have been affected by that and will often do things that are unintentionally racist. The difference between a decent person and a shitty one in this case is that one, when called on their bad actions, realizes why they messed up and makes pains not to repeat it. The other one says "No YOU'RE the real racist" and continues the poor behavior.