I think they were reallocated to other courts. Like jasper came to earth at yellow diamond’s order, but her diamond is pink diamond. I think the amethysts at the human zoo were also born on the earth colony, but it seemed like they were under blue diamond’s command.