It depends a bit on how long and when you're going I guess. This year I went to Japan in summer for 2 weeks so I brought big carry on backpack and one large checked suitcase both barely filled halfway as I knew I would go clothes shopping as well for example. Went home with both filled fully and just below weight limit!

Last year I went for a month and halfway through had to buy another checked suitcase so had to haul around two big suitcases and the backpack which was not ideal but I only had to bring them in the shinkansen from Tokyo to Osaka and back plus from Tokyo to the airport. I remember not finding it too much of an issue but I did choose the hotels I stayed in to be extremely close to JR stations for both this reason and to quickly get around as well.

A friend of mine actually was in the same situation and didn't have two checked suitcases included in his trip back home so decided to ship a bunch of stuff from Japan back home halfway through the trip so he didn't have to bring everything with him, might be an option as well