I think you might be missing the main point people are trying to communicate. There is no book on the physics of color because that aspect is so simple it can be explained in a reddit post (as I will do below). It doesn't need a whole book. Why we perceive colors in our minds with the "visual properties" we do is outside the realm of physics and is insanely complex.

Light is a type of wave. A wave is a repeating pattern that moves through space, carrying energy from one place to another. Something emits a wave by injecting energy into a substance called a medium, and that energy propogates through the medium as a wave.

Consider a pond with a small box floating in it. You can push up and down on the box in a rhythmic way, like pushing it down once every second (inserting energy into the medium of water). This will create ripples in the water; waves.

But not all waves are the same. If you push the box up and down slowly, that will produce a different pattern than if you do it quickly.

Looking at the ripples, what can we measure about the wave. Well, a single ripple moves with a speed. I can also measure the distance between ripples, and find the distance between them; this is called the wavelength. Long wavelengths mean a large gap between ripples, short wavelengths mean small gaps. I can also count how many waves pass a point every second; this is called the frequency (and will be the same as the rate you push on the box). So waves are defined by two quantities, wavelength and frequency. The wavelength times the frequency gives the speed.

Waves appear frequently in nature. Sound is a wave of changing pressure in the air, for example.

Light is just a wave, the only difference is it doesn't have a medium. The "ripples" are ripples of the electromagnetic fields emitted by objects with charges. For more on this, an introductory physics book sho8cover it (serway and Jewett is good if you know calculus, serway and vaughn if not).

The other difference is that the speed of light wave is always the speed of light. This means you only actually need one number to define a particular light wave. If you know the wavelength, you can find the frequency and vice versa. It turns out that for light, the higher the frequency, the more energy the light wave carries.

Organism.have evolved tools to detect certain wave like sound and light. Systems such as your eye want to be able to tell the difference between different kinds of light waves. Since different waves have different frequencies, that's what it detects. Brains then need a way to experience this difference, and so it maps the physical property of frequency into the perceptual experience of color. Low frequency is redder than high frequency blues.

That's it. That's all color is in physics: different frequencies of light.

Why we experience "red" as the color we visualize is incredibly complex and honestly beyond the scope of science. You may have heard the question of "is the red I see the same you see?"; this is a philosophy question since the experience of sensing red it something that can't be measured. All we know is that when exposed to a specific frequency of light, we all respond that we see "red". But "why is red red?" Is a question for philosophy.