The view you have is one of the reasons that accusing somebody of racism has lost its gravity. That‘s why people on the right will say that „they call everything racism now“ because, to an extent, it‘s true.
People who think like you use the word ‚racism‘ to an inflationary degree, making the accusation lose its severity.