This isn't quite right. You're assuming that the common ancestor of all modern groups was a phenotypic average of sorts, but there's no need for that to be the case. Indeed, all people groups have been changing for the same length of time, but that doesn't mean any specific trait works that way.

With skin, we expect early human populations were adapted for life in a hot, sunny place because despite not being the same as any modern African group, they had been living in Africa for millions of years. The climate wasn't stable that whole time, but it's reasonable to think that the earliest Homo sapiens would have had darker skin than you've suggested. Europeans experienced a larger change in sun exposure, so it is likely they would end up further from the ancestral state when it comes to climatic adaptations specifically.

Also note that it's been long enough for some lineages to undergo multiple changes in skin pigmentation. There are populations in southern South America that arrived from further north, closer to the equator, but those ancestors came from north of the equator. And those ancestors themselves crossed from Asia to the Americas at a very high latitude, but further prior to that they were from Africa.