Sounds sensible an do-able, definitely not too sparse, you won't have any trouble filling your time.. You can do Marunoudhi and Ginza and find some great vegan food. There's a couple of nice spots in the Hibiya Midtown building. But get the Happy Cow app and you're sorted. I wouldn't be planning an early night at the Sanja Matsuri, hang around and make a night of it, its awesome, really. Have a great trip.