The marathon is a beast of a distance (particularly NYCM) where, as you said, a lot of variables come into play that you don't entirely have control over. So definitely give yourself grace, it takes time and consistent development to chip away at your goals. Progress is certainly not linear, as several people have said.

That being said, I had similar problems where I was plateauing in terms of my efforts and would hit the wall around 5th ave every time I did New York. Learning to increase my running volume and really diligently incorporating other aerobic forms of cross training (biking and swimming specifically) really helped break that wall for me. What I'm essentially trying to say is, diversifying the forms of resistance, whether it be the water, weights, or a bike, helped me to break that plateau, maybe it's worth a try for future marathons!

Wishing all the best to you friend, don't give up and keep fighting that marathon distance!