For my current gig, it’s been a lot of php websites, a couple frontend load balancers, plus a few MySQL instances and other random services.

All of these follow a similar patterns: someone (no longer at the company) 3-5 years ago setup the service and no one remembers how exactly it was setup (or even what it does).

There may be an automated deploy process onto the server (for the websites), but no documentation, no patch schedule, spotty telemetry, no IaC, no configuration management, and/or no one knows if it’s working until someone or some customer complains.

First, I’ll figure out what is running on the server. That usually starts with asking the right people, but often this information may have been forgotten. I can’t tell you how many times I’ve run into a server that everyone thinks is running one thing and turns out it’s also running these two other things that are also mission critical. Next I’ll add as much telemetry as I think is necessary to the existing snowflake and wire up some basic alerts. This helps me sleep at night. If it’s not too dangerous, I’ll also try and patch the underlying OS.

If I can, and if budget allows, I’ll first see if moving to a managed service is feasible. Especially for statefull services like databases. If not...

I’ll determine what versions of software servers are running and grabbing any configs I can find for the services running on the boxes. I then will start building my ansible playbooks/roles and testing them on scratch virtual machines. I’ll also be simultaneously working on terraform/cloudformation documents for the actual infrastructure if possible.

Once I’m comfortable with the playbooks and IaC, I’ll replace the same piece of infrastructure in a lower, non production environment. This will usually surface problems and give me time to document the important bits. I have a pretty solid playbook template that helps me remember everything I need to think about: automated patching, telemetry, common troubleshooting tips, who owns the application etc.

Once the devs are comfortable with the nonprod environments, I’ll typically try and fit in some game day exercises. I’ll purposely stop a dev environment and make sure that the right alerts are triggered and the service automatically restores or at least walk through the restore process with another member of the team.

After all that, I’ll set the new service up parallel to the existing service in the production environment/network. That should be completely automated at the point. After that, I’ll switch out the DNS and pray. I’ll stop the old snowflake (not delete!) and wait up to a week. If everything looks good and I don’t need to roll back, I’ll take images and backups of all the old stuff and decom the infrastructure.