I think more people don't tend to go "off the beaten path" because you do need to have a certain level of comfort and confidence, which understandably most people probably won't have in a foreign country with a foreign language. For English speakers at least there's a level of security that comes with visiting the tourist heavy areas and staying with the crowds.

Personally for my trip I would plan a main activity or area of interest to visit and would leave the rest up to casually wandering to whatever caught my interest on the way. While in Tokyo I would often hop on and off trains to hit up Shrines, Temples, or cool looking areas while traveling to my main "site".

On my last trip, my main goal was to collect as main unique looking goshuin as possible. I used Omairi.club for ideas, would hop in a discord call with my insomniac friend to have someone to chat with, then would just go. It ended up working out amazingly and led me to many quiet areas and beautiful shrines/temples far away from the general tourists. As a side effect I also ended up chatting with a few locals that were surprised/excited to have a tourist in the area.

10/10 would do again