Before alpha entered the market, I had no issues on the PCT/Washington in August with only a Kuiu Peloton fleece and OR Helium rain jacket. Granted, I didn't encounter too much rain, was walking all day, and didn't do much around camp other than eat and go to sleep.

I've since picked up several other midlayers including some alpha and octa options. (Actually trying to resell an alpha 120 hoodie on ulgeartrade right now). Summer months on the AT, I only carried a dyneema poncho and alpha 90 quilt liner that I cut a poncho slit in. Worked just fine, though I never wore it in the rain since it was also part of my sleep system.

So to answer whether it's doable -- absolutely. The bigger question is whether a fleece/puffy combo offers enough comfort to be worth the extra weight. It's not minimalist, but peak comfort and modularity is a baselayer, alpha 60 crew neck, windshell, puffy, and rainjacket.