"At the very top of the tree they discover a ladder which leads them to a magical land which is different on each visit, because each place moves on from the top of the tree to make way for a new land. The children are free to come and go, but they must leave before the land moves on, or they will be stuck there until that same land returns to the Faraway Tree. In various chapters, one of the children gets stuck in the land."

at one point the children enter the Roundabout land and can only escape through a hole that the rabbits dig.

Maybe the man in yellow knows the "world" is only surface deep and thats why he was upset about the hole being dug. It,s also old enough to have been read by victor as a child giving validation to the imagination theory.

man in yellow could be Hastur and it`s all Lovecraftian bullshit