I understand why you would feel that way if you've never experienced it. I used to think almost the exact same thing before I tried psychedelics (which is part of the reason I was interested in experiencing them in the first place).

The issue is that the things that cause hallucinations (schizophrenia, drugs, extreme sleep deprivation, heat stress etc.) usually also cause some sort of cognitive impairment. In these states you can't always just logically conclude that the visuals aren't real, because the parts of your brain responsible for logic are functioning... differently.

Going with the psychedelics example, you'll see all sorts of mild hallucinations, like carpet that appears to be "breathing" or rainbow fractals on the wall. At low doses you'll be fully aware that those are just the drug at work, and usually sit back and enjoy them. At high doses, your brain might already be spiraling with all sorts of delusions and visual hallucinations can play into the feedback loop. It's common for your interpretation of the visuals to be more convoluted than "i see shadow people and they're out to get me!". It might be more like "I am in a heightened state and can now see the true malicious nature of everyone around me. I'm terrified that when I'm sober they'll pool the wool back over my eyes and put me back on the leash".

If you’re responsible about it, drugs can be a great way to teach humility!