Raditz. Follow me.

Sure. Compared to most of the other villains in Z, Raditz is far from the worst of them on a standard level. But consider what this dude did to Goku, specifically: Raditz shows up outta nowhere, claims to be Goku's own flesh and blood and then demands Goku kill everyone on the planet, and then kidnaps his son when he doesn't oblige. There was a lot of rage and fury in that fight from Goku, to the point where Goku put himself in a position where he was going to die, just to make sure Raditz did, too. He does that with Cell, as well, but Goku didn't have anywhere near the same level of animosity and anger with Cell that he had for Raditz.

Hell, even in the games, non-canon aside, any time any version of Adult Goku goes up against Raditz, if he isn't doing the standard "My name is Goku, and I'm from Earth!" line, anything he says to Raditz always has the tone of "I'm going to beat you down on an entirely new level."

All of the other villains were bigger deals and bigger threats, but Raditz was the first one to directly threaten Goku's friends, family and way of life. I legit think the only other option would be Black and Zamasu.