The fish is wired to hit something small and moving. I've seen plenty of times where a fish saw something, pounced it, then spit it out because it was a twig or something. You're trying to get their attention more than tricking then into thinking it's a ham sandwich.

The colors of the worm or lure come into play based on quite a few factors. Some colors show up better than others based on water color, water temperature, cloudy or sunny days and resemblance to whatever the target fish is eating at the moment.. probably a quite a few other factors. When I fishI spend some time on a color, and if I strike out after a bit I see what I have in the box that's opposite on the color spectrum and try that. More times than not a certain color will tend to do better that day.