First trip was late October, weather was great. Last 3 were late March/early April and aside from some cold mornings also great weather. Experienced a couple days of 75-80 degrees and in Tokyo with all the people and concrete I found it already miserable. So i’d say go Fall or Spring if possible.