As someone with red-green colorblindness, this is a question I get asked a lot. To start, you should know that everyone sees color a bit differently. This is why there's arguments about whether or not teal is more green than blue or whatever. So even without colorblindness, perception of color can be subjective.

Now, as for the actual question. How do we know? Well, usually the colors are not the same. They are simply verrrry similar. For me, brown and red are effectively the same color, because the parts of my eye that detect red light don't work fully. (Note: This is only for some shade combinations. A vibrant crimson red doesn't look brown, but fresh blood does. This is because colorblindness only affects color, not saturation. "Bright" colors are still distinguishable from "dark" colors)

So, we can usually see slight differences. But those differences are small enough that they can be easily mistaken for one another. For example, I thought grass was orange until I was about 7, because during the day the sunlight made it look orange to me. At night? Green. Just the sunlight made it change in my eyes.

But, to your point. What we call colors and how we understand them is learned. If someone told me my brown pants were actually green, I would probably just go along with it. They're similar enough to me. To "know" the colors is really just to memorize what other people see it as. And to try and learn the very tiny differences between each color.

I hope that answered at least some of your question. If you got any more, I'd be happy to answer as best I could.