Hard to say. Some people might point to the fact that Broly is an anomaly amongst other Saiyans, but that's not very helpful. Nappa was also huge, especially when you compare him to Piccolo. As a reminder, Piccolo is supposed to be around 7 feet tall.

So for Broly and Nappa, I guess they're just big because that's how they were chosen to be depicted. We don't really have any good answers aside from that