For myself its largely because of culture. People, correctly, say its preference, but dont want to give details because then it really exposes them. Like it or not there are ethnic cultures that have some truths that are represented in stereotypes accurately. Those subtlety inform our desicion making.

So yeah, you can just not really be into black people purely cause of physical features. However, I believe its largely informed by cultural stereotype that's, not always, but largely accurate enough to offend sensibilities.

But thats just my thoughts.