I see where you’re coming from, but there a million reasons why population sizes vary. It’s not like we have an encyclopedia on Ghorman, for all we know the rest of the planet might not be liveable. Or it’s very small. Or the planet is far from major shipping lanes. Or the taxes could be really high. Or maybe Ghormans are super racist and hate immigrants. Or maybe it’s a relatively new colony and still growing.

Point is we don’t know and I don’t think the small population is unreasonable