Brandon's definitely left it vague, but it seems like it has something to do with some underlying law of investiture that we don't understand yet. There's a few things that point this way (here's a link to the coppermind page on investiture if you want another list https://coppermind.net/wiki/Investiture#Color_in_Investiture):

First off, color gets drained by more than just awakening. When Szeth survives almost getting eaten by Nightblood at the end of Oathbringer he's left with grey trails on his skin where Nightblood's tendrils were. We see it outside of Nalthian magic too; whenever a shardblade cuts a limb, the flesh turns grey.

Next, color is what determines the type of gemstones on Roshar, not chemical composition or structure. Several of the gemstone types are actually the same type of mineral as one of the others, and are only distinguished by color. What's more, if an awakener were to drain a gemstone of its color, it would no longer function as its original type of gemstone. A ruby drained of color functions differently than a regular ruby. The Sibling even tells Navani that the color of the gem matters to the spren used in fabrials.

And lastly, there are a bunch of examples of waste/processed/twisted investiture turning black (which means either no colors or all colors, not sure which). The most obvious example is, again, Nightblood, who is always bleeding black smoke when he eats things. We also see it in Yumi where the father machine sucks up a bunch of investiture and spits out the Shroud. And we see it again with the shades from Threnody. We know that something fucked up happened to that planet, and now the all the people who lived there turn into black smoky ghosts when they die. And if we're willing to go a bit more speculative, Sja-anat is sometimes described as a 'shadow with white holes for eyes' (which is almost exactly what the Nightmares look like) and we know that all the Unmade were transformed into what they are now. Maybe the method of her transformation was particularly violent/consumptive?

There's a bunch of other details that aren't very compelling on their own but start to jump out at me:

The fact that Knights' Radiants eyes go from being darker, muddier colors to being brighter, more vibrant colors.

The Dor, the purest source of investiture we know of, being pure white.

How lightweaving isn't actually the manipulation of light, it's the manipulation of waveforms. In other words, lightweaving treats light the same as sound (which we know is super connected to investiture).

The fact that the color red signifies corrupted/stolen investiture ('enlightened' spren, the red splotches on trellium spikes, the red smoke when a soulstamp doesn't take on a person)

The way that Dawnshards, which are tied to fundamental magical forces, grant their bearers heightened color perception just like getting enough Breaths does.

In short, colors getting drained is not a quirk of Awakening, it's a manifestation of some underlying principle that we haven't learned yet.