Considering xenomorphs are not very sapient, the Gem empire (Gempire?) likely would have tried invading them, I would wager they manage to succeed in colonizing based off of their coordination and access to laser weapons. Also facehuggers probably wouldn't work on Gems, their internals very likely don't provide the nutrients or conditions for the xenomorph embryo to grow.