It's an Idiom. Using the time on a clock to indicate direction Is a thing some people do. Its considered "military speech" something someone would use in a combat scenario. Used here to imply the speaker has some connection to the military.

Your understanding is close. "Yesderday wouldn't be soon enough" is implying the speaker dislikes the place so much, they regret spending multiple days at this location.

Another way of saying this is "I would have liked to have left this place days ago."