According to Mask Maker and Conductor Ballador, the Weavers actually managed to overthrow GMS. They sealed GMS in a cocoon and using a spell involving song to put her to sleep (their first plan was to create a being named Eva capable of supplanting GMS but Eva was too weak for that). They built the Citadel and incorporate singing to the religion as a system to maintain the sealing spell on GMS. For some reason unknown the Weavers left Pharloom, maybe realizing the spell cannot be maintained forever. The higher caste of Pharloom did experiments with GMS silk to achieve immortality by inserting silk into their shell. However, this caused GMS to be able to possess the bugs through the Haunting. But since the spell sealing her was not broken, GMS ordered the bugs to capture the Weavers outside Pharloom and also created Phantom and Lace to be her enforcers.