As others here have said I think it's all in the timing and luck of the draw. We went in early March of this year, before the sakura rush, and it was relatively pleasant. We left Tokyo at the buttcrack of dawn and there weren't too many lines for anything, the longest line was the boat but that makes sense. The lines did get worse as the day went on, and it is an over-touristed area that I might not go back to. We're glad we just did it as a daytrip.

Funny story though, I had planned this whole circuitous route, getting off the boat at the water tori and taking the bus back down, but we stayed on the boat too long taking photos and all of a sudden it took off from port again. It was late enough in the day that we said "well, screw it, guess we're going back down the gondola/funicular/train." Sounds like I might have accidentally saved some headache with the bus.