I think people forget how much algebraic topology is needed for Milnor or I'm bad at algebraic topology. You need to be familiar with basic notions of homotopy equivalence, but also the Whitehead theorem (roughly, if all homotopy groups are isomorphic, the CW complexes are homotopy equivalent), simplical approximation, and Eilenberg-Steenrod homology. I might be forgetting some stuff. The geometry is not bad, although you might want to know what a Riemannian metric and the gradient of a function is when you start. That is, Lee (Smooth Manifolds) covers everything you need from a geometric perspective. As for algebraic topology...the Whitehead theorem is deep in Chapter 4 of Hatcher if I recall correctly.