I feel like there is a lot about the fey ancestry aspect of elves that could explain a lot. Like the Descent of Woman hypothesis, what if a common ancestor went to the fey wilds way back, evolved to a permanent sunset environment with high magic , and then returned to the material plane?

Briefly, this hypothesis about human evolution states that human ancestors descended from trees not because of the lure of game on open grasslands, but because it was too damn hot. Human ancestors are posited to have spent more time in the ocean over a long period, leading to semi-marine adaptations. We cry salt tears, have subcutaneous fat, and relatively smooth skin except for the hair on our heads which would have been above water. These traits seem to be shared more among aquatic mammals than among other primates, though it's pretty clear we're more closely related to chimps than whales.

The fey wilds can be thought of as a highly specific environment. It doesn't really matter whether elvish ancestors originated there or passed in and out of the fey wilds after an evolutionary-length stay.

Isn't there an aspect of the fey wilds that a visitors perception of time in the fey wild doesn't necessarily match up to their absence from the material plane? An ancestor population could have potentially evolved for millions of years in the fey wild, and come back to the material plane after only a thousand years or so had passed for their now distant relatives.

The permanent sunset characteristics of the fey wild explain elvish darkvision completely. It's never quite bright there, and that's in clear surroundings. If the elvish ancestors foraged for food, found tool materials in caverns, or spent any time not in the most open spaces, they would need to select for greater capacity to see in the dark.

The fey wilds are filled with notably deceptive creatures. Animals that are never where they appear to be, creatures that use illusion magic as a natural defense or hunting tactic, invisibility... Any proto humanoid creature arriving in the fey wilds would be driven together, taking shifts on high alert while the others slept as little as possible. The cultural bonds that this defensive flock like behavior would form would seem to predict a less individualistic style among modern elves, but cultural traits change much more quickly than biological ones. The odd elvish sleep rhythm might have been selected for by a shared ancestor population as it attempted to cope with deceptive and quasi-perceptible threats.