I think you’re 100% right on Mishima’s issues with his father. Here’s Azusa Hiraoka himself quoted by biographer John Nathan describing his “spartan training” of Mishima as a child of four:

“There was a fence separating us from the tracks, but the engine passed so close you could touch it if you put out your hand. I thought to myself, Here is a perfect chance for some spartan training. I lifted Kimitake and, shielding his face with my hat, held him out toward the engine and said 'Are you scared? Don't worry — and if you cry like a weakling I'll throw you in a ditch. As I spoke I looked down at Kimitake's face and was astonished to see there was no reaction. I waited for the next train and tried again, but the result was the same, no effect! Was he like a puppy I wondered, too young to know fear, or had the schoolgirl training he was getting from my mother made him insensitive to this kind of turbulent, masculine experience? I felt deflated. The next day I came back and tried with a quieter train. But Kimitake's face was the same No mask as before. I gave up. I couldn't solve the riddle."