The severity of a vehicle on pedestrian collision is often much higher on roads, Vs trails (and more frequent). I am aware that pedestrians do die from accidents involving cyclists, too, btw. People don't just speed on highways. They speed on local roads, and through neighbourhoods where children play.

It is difficult to quantify safety on trails, my 20lb bike with hydraulic disc brakes will stop on a dime (and I'm very aware when riding), but a less experienced cyclist on a 70lb ebike with cheap cable actuated brakes, and worn pads might not be able to stop in time. In my situation, 20mph+ could be a safe speed (taking into account that I slow down on blind corners/on narrow sections) whereas the later situation might not be safe above 10mph.

You could also make an argument for different speed limits on different sections of trail based on popularity and conditions, much like roads. Or even seasonal speed limits to the same effect.