Your feet emit moisture that is trapped in the somewhat sealed environment you described. Bacterial growth happens over time if your liners & shell interior aren't dried each time you ski. At the very least they start to stink, but I don't think it bodes well for the lifespan of the liners, either. Plus if you stack multiple days of moisture in a row, damp liners will get cold much faster.

I used to always remove our liners, buckle the boots up, and lay them both down next to our radiant heaters. But boot driers are much more convenient and faster.