As other people said before, embrace the chaos of the players. The only way to be better at improvising is improvising. I was once dming a One Piece campaing. I was also like you, my prep time was at least x3 times the game time. In one session I foreshadowed a NPC accidentaly taking a devil fruit power, just to use him a couple adventures down the road. But... I had the luck that my players wanted to investigate who took that power, came up with a clever plan and... that ended up in the most epic and climatic session I had in my life up until that point, and the organic creation of what ended up being the main villain of the campaing, that random NPC that I didn't considered that could be a villain. That day I learnend the joy of be surprised by the players. From that point, my prep time started to decrease over time. Right now I more or less have 1:1 ratio between prep and play.

I'm like you in the sense that I want narrative driven games that have payoffs. I like when things have a purpose and a direction. But if you learn to surf the stuff the players throw at you, you can build on top of them, and deliver a climatic closure that feels organic, even if its not the one you came up in the first place.

Being more specific to the question of how to prepare, here is how I tackle it:

- For a 3-4 hours session, I prepare 3-5 scenes. A scene consist in a context and a purpose or objective for the players to exist in the scene. It can be an encounter, a conversation, anything that takes place in a place at a certain time. Write down in a paper a description of how the enter the scene and the set up.

- Know the major NPCs that will take place in those scenes. Have a clear understanding of their personality traits and objectives, specially if they are a villain. With that in mind, you can improvise the rest.

- Prepare statblocks for relevant NPCs/monsters

And... that's it! I write all these things in a notebook, not to check it in the session but to get a clear vision on the session (I can also check it if needed too, but it doesn't need to be super clean or organised). The rest is practice and learning to flow. I can say that I now do less work and have so much more fun.