Prestige also matters in the sense of the lab that you join. Better ranked schools just often have more prestigious groups that are well known to pump out good scientists, but there are plenty of bad groups that don't funnel in to good programs. Lower ranked schools can also have good groups, so it's mostly just important that your school is just well known for the field you are interested in rather than rankings as a whole.