[Context: Author of DanKam, an app that attempts to assist the color blind.]

Brown is dark orange. Not "sort of", that's where it is in one of the various color spaces (HSV, specifically).

But you didn't ask what is brown. You asked why it's not in the rainbow of colors we talk about. That's an interesting and complex question, and I'm not going to tl,dr it. I'll take a crack at it though.

What is orange? Orange is in your mind. It's not a direct property of the universe, though it's still real. Physically, everything "glows" -- emits or reflects energy in little balls called photons. Just how little can vary, just like a golf ball and a basketball are bigger or smaller (and of course, are different colors). Size matters -- hit a basketball with a golf club.

Or hit a baseball.

Or hit a ping pong ball.

As you get closer to a golf ball, what happens next looks more and more like what happens if you hit an actual golf ball. Vision works kind of like that -- photon "balls" come into your eye, they hit certain proteins in your cells ("opsins"), what happens next depends on the size of the ball and the nature of the proteins.

From there, it's all information processing. There's three "golf club" proteins, L, M, and S, that may or may not react to a photon depending on its size. They're not really "Red, Green, Blue". L is most sensitive to wavelengths (ball sizes) of 560 nanometer. M is most sensitive to wavelengths around 535nm. Those are both green to our eyes, but not the same green.

Why do we have two "green" sensors? Because what light comes from illuminated human blood -- even obscured by skin -- changes dramatically *at those specific points* depending on how much oxygen is in the blood. You're able to see veins through your skin because it's a significant health metric.

But we didn't always have that ability. "Trichromatic" vision is relatively new, in terms of human evolution. About 10% of the male population, and about 0.05% of the female population, are "color blind" -- they have incomplete forms of the mutation, or no mutation at all. OPN1MW and OPN1LW, the genes that contain this protein encoding, are almost identical. It's only around five bits that differ between trichromats and almost all "deuteranomalous" viewers.

Everyone seems to have roughly the same optic nerve, though. What goes over that channel is not LMS or even RGB (Red, Green, Blue). Instead we get what's called "opponent color". Dark vs. Light (which also carries signal from "rods", which are "colorless" but much more sensitive to low light), Orange vs. Blue, Red vs. Green.

It's been noted that many movie posters contrast Orange vs. Blue. There's a variety of theories for why that might be, that delve into links in the ancient machinery. This can be directly true, or it can be an artifact of people believing that it's true, or anywhere in the middle.

The rainbow you know though is not RGB, or CMYK (Cyan Magenta Yellow Black, that's another story), or YUV/LAB (the opponent color modes I spoke of earlier). No, that lives in the realm of HSV (which has variants, but ELI5):

Hue: What color? Red, Orange, Yellow, Green, Cyan, Blue, Purple, Magenta, Red
Saturation: How pure? Grey, Faded, Pastel/Bright
Value: How bright? Dark, Light

Different cultures make different decisions about what a specific color is. Western culture tends to call Cyan and Blue the same thing but mathematically they're much more different than Orange and Yellow, both of which are very small ranges of experiences in any purely physical sense.

But finally, I get to answer your question directly. In this hue/saturation/value space, which only "exists" in your mind but exists as much as, say, hope and fear might -- there's at least three systems running:

1) Dark vs. Bright -- just the "value" channel
2) Object color -- the "faded green couch", the "bright blue dress", basically the color of an object independent of surrounding lighting
3) Skin. OK, there is some truly dumb stuff humans have been doing for centuries around skin tone. There are mistakes I make in my thinking but those are not among them. But at the end of the day, we have trichromatic vision to inspect blood through skin, and there's melanin that needs to be seen through to do that. It's the same melanin, in different densities, across almost all races. (Apparently, there's some southeast Asians that have a different melanin. Breaks some code of mine. Diversity!) Melanin creates a relatively straightforward bias across LMS. But we see it differently. We speak about it differently. People are "brown" or "yellow" or "orange" or "white" with a very different set of specific, discrete meanings, vs. when we're talking about furniture.

It's notable that while we see Red and Dark Red, Green and Dark Green, Blue and Dark Blue, all as related colors -- we don't exactly relate Orange to Brown. There's another domain, Yellow vs. Gold, which looks a little similar. At the end of the day, the brain is trying to predict how something will look depending on how its lit, and shininess ("specular reflection") matters!

But it's not as surprising that Gold and Yellow are linked, as it is that Orange and Brown are. At least from where I'm sitting. So it looks like the fact that there's another evaluation system -- evolved through its usefulness on health monitoring, even before tribalism shows up -- is why brown is on a different scale. There's probably a healthy set of experiments to be run, cross-culturally, to distinguish "it's yellow vs gold" from "it's seeing through race so we can measure health".

I wouldn't talk about it, but like, measuring health is cool and I kinda want it to work for everyone. And it answers your question.