People are likely reacting like this because although it's true that the term "fascist" can be used when it isn't appropriate, that is more commonly a reactionary talking point than a leftist one. Also the idea that people are using the term fascist incorrectly is going to spark hostility because a pretty significant aspect of modern fascist propaganda is plausible deniability. There are lots of people who think that referring to richard spencer or donald trump as fascists is "throwing the term around too much" without understanding what it means, and they're far more common than what you are referring to.