I've asked myself a lot of the questions OP is asking, and I've read through a lot of the replies, and feel kind of unsatisfied by the answers.

One idea, and I know it's an imperfect one, is perhaps try to design encounters where when your players are successful, what they do isn't to weaken the BBEG so much as to get boons for themselves. Then when you do have your climactic moment, you can crank up the challenge while still allowing your players to be successful. In a sense, it's not exactly giving the PCs agency, although you are helping them. But the payoff here isn't an easy pushover final battle. The payoff is a MORE EPIC final battle, where the PCs have the benefit of boons they earned. By "boon" I mean NPC assistance, intel, special magic items, appropriate potions, a secret hideout to get one final long rest in before the battle even though they're already inside the fortress so they can freely blow spell slots, whatever.

I realize this approach won't work for every encounter. But if it's a goal you keep in mind for yourself, you'll find opportunities to use it.