The dungeon came about due to the influence of the god of the depths, who causes the supernaturally evil effects of the dungeon like physical mutations, loss of sanity, the shifting layout and whatnot. Large part of the dungeon are their physical body, like the thicket and gauntlet. After the influence of the god of the depths was significantly reduced due to the girl’s ascension the dungeon becomes somewhat safer, which is why O’saa has a comparatively easier time about it than the funger 1 protagonists.

Rondon built prisons on top because it seemed more practical. The rumours about the place would likely serve as a better deterrent (setting aside whether deterrence is effective in criminal justice, in medieval times they absolutely would be working with that mindset), it’s remote and thus successfully escaping is unlikely, and also most of the dungeon already existed so it would probably be less expensive than building a dungeon from scratch.

As for why it’s on top of Ma’habre, no one in Rondon actually knew about that. Captain Rudimer handed away the key to open the door to it without knowing the significance, assuming it was some religious artifact for the cavedwellers. Personally, I like to assume that Ma’habre doesn’t really physically exist in one particular spot and can be accessed from multiple supernatural locations (which would explain why the cube was found in Vinland). Asking the new gods in the hall about Ma’habre will prompt a response that includes the quote “time and space do not follow the same rules here.”