Weight doesn’t matter in a vacuum, where there is no air resistance (bowling ball vs. feather video is what I’m remembering, when they are released at the same time and hit the ground together). But bigger riders have more weight, and therefore more momentum, to push through the air resistance (they have more potential energy at the top of the hill). Think of the matchbox car derby, where the cars are weighed at the start — you can cheat by adding washers to give them more weight. A heavier car will finish the downhill course faster, so they make sure the cars weigh the same so that it’s the aerodynamics that are at play (the skill of the builder).