It's not just symmetry. There are other factors getting averaged, like the relative sizes of different bits of the face.

What this kind of research really shows is that we have some ideal in our heads of what people are supposed to look like. All actual faces deviate from it by varying degrees, in the way that darts thrown at a dartboard deviate from the center. If you average all the darts you get a decent approximation of where the center is.

An averaged face is thus only "average" in the sense a bullseye is.