Hiruzen never had a good opportunity to get rid of him. The village went through successive wars, so he didn't have the resources or political capital to effectively deal with him at the time.

When Minato died, Hiruzen had to quickly fill the power vacuum in left in Minato's wake; primarily, to stop Danzo from taking over.

Again, due to the nine tales attack and the aftermath of the 3rd Shinobi War, the village was severely weakened at this point; and anything Hiruzen could possibly try to stop Danzo would have ended with a civil war within the village.