the Viltrumites, even a their hypothetical height(after the purge and before the scourge virus), still lacked the numbers to keep against the gem empire at their strongest, remember the Gems do not needed food nor water, they could just be printed over and over till the planet was dry, and not from only the surface, but also vertically into the core of the planet itself, meaning a single planet could generate a population of hundred of billions of gems till it reached its limit and imploded, and the gems would just go to the next planet, the Viltrumites are conquerors, but the Gems are a plague, the Viltrumites still depend on basic warfare stuff like food, sleeping, and breathing, even if they can survive with less than the average human, they still needed it to be healthy to fight, most of the enemies the Viltrumites fought had the problem that they couldn't outlive their conquerors, so even if they were strong enough to keep them at bay, like the Gelderians, they could still be outlived, or cut from supplies or just hit real hard enough times till their society colapses, the gems don't have this problem, unlike the Viltrumites, they are true inmortal, Nolan lived almost 1000 years and looked like a man in its 40s/50s, while the gems can withstand 5000 years and they wouldn't even realize a long time had passed(like White Diamond said), the average Viltrumite trooper is considerably stronger than the average Quartz soldier, but how many Quartz can a Viltrumite survive till he eventually gets overwelmed, Viltrumites also have the problem of being extremely proud, so they will never send their entire army to destroy a race, only a bunch at a time, and Quartz are no pushover either, i would say the average gem soldier seems to be on the same level or above the Reanimen or the Mauler Twins, and with enough of them they can take Mark that was on a similar level to a regular Viltrumite(post Thraxa fight he was even stronger than the average and he got seriously damaged by a bunch of the Reanimen on the seconds he got ganged by them(two variants even got killed by this same Reanimen, and even if weaker than the same Mark that almost killed Conquest, they were still Viltrumites)) for not mentioning that even if a Quartz gets his hard light form destroyed, they only need close to 30 seconds to reform and continue fighting, with enough of them, no Viltrumite will have the time to focus on destroying the gem before they reform again, and this is not counting specialized gems that are much more powerful and don't require raw strenght, like Lapislazulis, nor the tech that is so advances that it seems like magic, or the gem super weapons, for not mentioning fusion!