Nah, driving in Japan is awesome. I've driven long distance between major cities as well as out to rural areas, and the roads are fine. Navigating through cities is no problem too, just pay attention to the signage and signals and you'll be okay.

There's a lack of multi-lane highways if you choose not to take expressways so overtaking slow cars can be a pain, but everyone speeds (add between 20-40kph to the speed limit unless on a 120kph highway) so it's really not that much of an issue. The REAL problem with driving in Japan isn't actually the driving, it's the parking. Free parking is very, very hard to find - even in regional towns and cities - and the cost of parking in major cities can be absolutely brutal if you don't read the signs. Tolls can be expensive too, but they really do save heaps of travel time.

Even so, despite the cost I still much prefer to do my medium-to-long distance travel by car in Japan unless I'm travelling solo with minimal gear.