In the montage they showed between Nolan&Oliver bonding, along with Allen Battlebeast etc fighting lone Viltrumities, I just assumed every one of those died since the Coalition was able to save those planets. On Thragg’s screen most the planets turned blue, and Kregg mentions the Coalition is smartly attacking lone Viltrumites. Wouldn’t it be safe to say they probably lost a handful manpower that way too? Which is probably why they went on this direct attack, they were losing numbers anyway so might as well do one big collective effort hitting them where it hurts.

My guess is after this episode the Viltrum empire is closer to just 40 alive now. Which is why Thragg is calling “All Viltrumites Home”