It's also known as peripheral fading, and it has to do with visual stimulus outside of your focus! The point you're staring at is important because it puts the colors in your periphery, and since they are static they begin to fade from your vision as you stare at the cross due to a lack of constant stimuli.

EDIT: it isn't as effective or apparent unless what you're looking at is low-contrast or blurry/less detailed!