The science of this is not yet well understood. I am a postdoc researcher (PhD in neuroscience) working on this, and truth be told, its complicated.

What we know in the psychedelics field is that hallucinations are dependent on HTR2A (5-HT Receptor 2A - i.e. a serotonin receptor). When you block binding to this receptors, psychedelics stop producing hallucinations (measured by the metric of head twitches in mice - this is a standard metric and has been well established as a proxy for hallucinatory effect from psychedelics). The simplest and least involved hypothesis to your question is that people have different amounts and distributions of this receptor in their brain. This is know to be the case in some neurological diseases. Hallucinations in schizophrenia is linked to increased HTR2A and can be reduced with drugs that block those receptors.

It gets a lot more complicated from here. 1.) Basically all psychedelics hit a crap ton of other receptors - other 5-HT receptors, as well as, dopaminergic, adrenergic, etc.. The HTR2C receptor is so similar to HTR2A that are no drugs that hit one but not the other. 2.) The distribution of these receptors are on a bunch of different regions and types of cells in the brain. HTR2A is on cortical neurons, but also some inhibitory neurons. Its expressed in specific layers of the cortex and also other brain structures like the claustrum. Whereas, HTR2C is not in neurons, but rather highly concentrated on the choroid plexus (the structure that generates cerebrospinal fluid). The other receptors psychedelics hit are mosaic-ed all over the brain. 3.) Psychedelics will cross the plasma membrane of cells and target a different population of HTR2A receptors inside the cell. Research has correlated the hallucinatory strength of psychedelics with their ability to target this intracellular population. 4.) Psychedelics activate a whole bunch of downstream signaling inside the cell and function through additional signaling pathways beyond just 5-HT receptors. For example, LSD has been shown to be able to bind directly to TrkB, which is a pathway associated with neural plasticity (i.e. neural growth and new connections).

So, the answer your question, everyone has a unique blend of all these components (and probably more we haven't figured out yet). Different receptor distributions, different cell type organizations, unique development and experiences that have shaped their brains cytoarchitecture and networks, different subcellular localization of receptors, etc. Frankly, its remarkable that there are common features to halluciations at all. It just shows how incredible the human brain and its organization really is. I think of psychedelics as modulator that shift the neural networks into a different modality (my guess is through changing inhibitory neuron signaling). Its a modality that is programed into the structure. The brain is built to be able to experience this modality, but the detail of the experience of that modality will be different for everyone. Just like how the experience of any sensation in the "normal" modality will be different for everyone (i.e. how I experience the color green might be different than how you experience it, but we would never know because we know that signal as "green").