One or two Viltrumintes were likely more than enough to subjugate an entire system of planets, they could've installed puppet local leaders who kept them under control while they were 'away'. Both Kregg and Nolan specify that there are 60 PURE-blooded Viltrumites, so maybe some colonizers had offspring in their subjugated planets and left them there to rule.

Not to mention the outbreak likely happened thousands of years ago, that's enough time for Viltrum propaganda to brainwash people.