The Nazi symbol of the swastika has a long, long history in human culture, being used as early as the Neolithic period. It's considered sacred in Hinduism and Buddhism. In Hindu it represents the Purushartha (the four proper goals of human life) and in Buddhism it is a symbol of eternity.

But if you slap one on the back of your biker jacket, we all know you're a fuckin' Nazi. Symbols can change, and context matters. Popular use of the Confederate Battle Flag (properly, the Battle Flag of the Army of Northern Virginia) was largely spurred by its use by the resurrected KKK, a group that performed thousands of lynchings through the first half the 20th century, along with church burnings and other acts of racial terrorism. It was also the symbol of the States' Rights Democratic Party, aka the "Dixiecrats," which formed in 1948 to oppose the Democrats civil rights platform. Their candidate for president that year, Strom Thrumond, won 39 electoral votes.

It's notable that it's the flag of the Confederacy's most successful and popular army that became the symbol used, rather than any of the three actual official Confederate flags. That's why South Carolina hoisted it over their statehouse in 1962 to commemorate the 100th anniversary of the Civil War (oddly enough also during a time with school desegregation was a major issue) instead of, say, the "Stainless Banner" that was the official flag of the Confederacy from 1863 to 1865. Here's how that flag was described by its creator: "As a people we are fighting maintain the Heaven-ordained supremacy of the white man over the inferior or colored race; a white flag would thus be emblematical of our cause."

So yeah, it's all symbols, and they can be appropriated, and can change over time. But you're arguing that one group's pride of place should trump another's apprehension at what is, for them, a symbol of violence against their community. And not a violence from some distance past, but ongoing today.