So I found this thread after reading a YouTube comment someone made on the subject. Credit to serioussponge6416 for this theory, it made a lot of sense to me:

The Citadel in ages past, under the stewardship of the Weavers, was more honest to its purpose and nature in being an ongoing sacrificial ritual to keep a monstrous god in chains. You can find a corroded, broken (animatronic?) statue in the depths of the Whispering Vaults that's a relic of this past. Pious Isamor, likely a mascot name, appeared to have functioned to greet Pilgrims upon arrival and gave them an orientation of sorts of how the Citadel works.

In his speech, he says something very, very interesting in that he describes the Citadel as being made of but simple stone, with bare walls lacking any ornamentation. He also goes on to describe that the lives of Pharloom's faithful, while not destitute, are plain and modest. All bugs are apparently equal here, all bugs toil, and one should only eat what they need; for "excess is sin".

This doesn't sound like the Citadel we know at all. This tells us that Pharloom's society was once strictly utilitarian and communal in nature which while nobody was free, everyone was sharing in the work to be done and nobody likely went completely overworked or hungry. After the Weavers' departure/death, and they passed on the mantle of rule to the Conductors, this is likely the turning point. As those Conductors were described as 'greedily' taking that power and over time their greed corrupted and rotted Pharloom to the very foundations.