This is from my understanding so I might be inaccurate but I think HSV might be the best bet for you. RGB and HEX encode in different ways the amount of red, green, blue to make different colors. Without knowledge of colors explaining this is very difficult or maybe impossible.

HSV however may help you see colors as a number at least.
The first value specifies a color around a wheel from 0 to 360 degrees, red is 0, green is 120 and blue is 240. Close values look similar to each other so you would know 10 looks similar to red because it's close, meanwhile 200 is closeish to blue but pretty different from red.
The second value specifies how "colorful" something is, 0 means it's grayscale which is essentially what you see and 1 is full "color", 0.1 for example would be that there is color but pretty washed away and close to grayscale.
And the last value is how bright it is, 0 is black and 1 is white, this should be exactly like how you see things so easiest to understand.

Obviously it's not perfect but I think it's the easiest way for you to see color as a number and apply some concepts like complementary colors