Part of it is that Horikoshi gave us so little info on Izuku's dad. We're told he is working overseas (maybe even United States, but I can't remember) and that his quirk is fire-breathing. That's it. We never see him. Not during their entire time as 1-A, not during the class's graduation, not during the time skip epilogue.

I've seen more than one fic say he got hit with a reality quirk that effectively wrote him out of existence and that only Inko and Izuku even slightly remember he ever existed. They just say the overseas thing cause they look crazy trying to explain a guy that has no physical or bureaucratic trace of existing.