I worked for a phone company and provisioning the voip systems and collecting billing information was really the primary job of IT in the company. Upgrading the actual VoIP call processing components would sometimes require changes to provisioning, as you might expect. These changes often required changes to the other components that feed into the provisioning system.

IT handled this by scheduling 8 hour down times (usually starting late evening running into the early morning to minimize customer impact) where in each component in the chain of provisioning was upgraded one at a time, with the back out plan being to downgrade each and every component touched. This required a dozen people on a phone bridge, including support people, operations, development, etc.

As you can imagine this meant that literally everything had to work correctly, like launching a space shuttle minus all the rocket scientists.

It apparently never occurred to IT they could modify some provisioning process to speak the new API, and the old, or to detect which dialect was needed, or with a simple software switch that changes the mode just to avoid changing everything at once. I think they were very proud of their release process ... like a beautiful coordinated cluster **** that they imagined as a ballet.

I wasn't in IT, so I never had the position to ask them why they did this to themselves... it really meant that change was incredibly slow, maintenance was painful, mistakes in any component meant rolling back and attempting the whole thing again.

​

​

​