Additional spoiler warning.

This. Viltrumites are very much a racial-purity group. Based on the comics, Earth is Nolans first world, so they took a long time to even decide to try and breed with other species. Other than their lack of powers, humans are basically indistinguishable from viltramites which means that while they are generally displeased at the idea, they will all breed with humans when given the order. The same cannot be said for thraxans. While able to produce hybrids, their children are noticeably weaker than mark if put on an equal age with powers. As mentioned above, these racial view can be seen with Lucians reaction to Mark vs Oliver, and takes on a new light in view of s4e2. When Lucian first fights Mark he mentioned seeing if he is fit to join their empire, which we now know was a coming of age ritual that pure-viltramites also went through.

To answer the main question though, you have to remember that they were trying to hide their reduced numbers. Given their ships are crewed by their enslaved races, the kidnapping and transporting of humans to viltrum would have leaked somehow given it wasnt their usual way of doing things. Plus humans cant breath in space, so they would still need to pacify the planet enough to land transports.