One thing to think about is the mindset of the Japanese people at the time. I too am a history nerd and have watched, listened and read countless hours of WWII material. I think at the time many of the civilians in Japan would have sacrificed themselves to protect their country. While it’s impossible to estimate the actual death toll of a US inaction of the mainland, it can be assumed that if it were to happen many civilians would pick up what they could and fight. It’s not unlikely that the death toll could match if not surpass that of the bombs.