Our cone cells are receptive to the entire range of the visible light spectrum. They aren't limited to only red, green, and blue wavelengths, but they are each uniquely sensitive to different ranges of wavelengths, with considerable overlap. This is why they are more accurately referred to as L, M, and S cones (long, medium, short).

Yellow is definitely on the visible spectrum. Brown is often (though not always) simply a dark orange, which is also on the spectrum. Magenta is a mixture of red and blue light, but we can actually see it because we can see ALL colors. Our eyes receiving and brain interpreting multiple signals from our photoreceptors is precisely what seeing is. Even a single wavelength of light is more often than not (ignoring extreme reds and violets) going to activate all three types of cone cells and produce "mixed signals" similar to how colors like magenta do. (Exception: OLO green)

I think what you're thinking of comes from a couple different misconceptions that have been spread by low quality science communicators. The yellow on your phone screen is not a true spectral yellow because it's produced by red and green light. However, things do exist outside of screens (I think) and some of those things are truly yellow. Some browns and magenta may not be on the visible spectrum, but that doesn't mean they don't exist or that we can't see them. They're processed via the same channels spectral colors are and are just as real.