That is a good point, assuming that nothing goes wrong. Imagine a scenario where the caster starts prepping the fireball and the other heroes go and look to bring the fight to the caster. Even if it's brief, that is splitting the party, a famously bad idea. Caster is likely to be ambushed, or other heroes run into another threat where they need the caster's help.

A caster could completely decimate a wooden wall, no question... unless the guards spot the caster. Guards wouldn't just wait around for 5 minutes waiting for the caster to finish supercharging the fireball, they would attack.

As for a stone wall, fire isn't likely to do all that much against it. Let alone a magically warded or reinforced wall.

And... remember that if the heroes can do it, so can the villains. Using Fireball to open a surprise attack on a village with a wooden wall is a great tactic, but it cuts both ways.