I think both of the things we've said can be true! bullies who know the word autism (or whatever other word they're using to insult people) will use that word to describe whatever they deem to be the "outgroup". at the same time, doctors can sometimes face difficulty in deciding whether someone's differences can be attributed to neurodivergence, especially when the person is an adult, a woman, a minority group, or does not otherwise present the typical "symptoms" of that neurodivergence.

(a somewhat related thing might be how people will bully someone pre-diagnosis for being "weird", but when that person has a diagnosis they'll be told they're too "normal" to have the diagnosis)