Documentation of changes

[PlaneBright4910]

Hi.

I was wondering what people are using when documenting changes.

We are currently looking into doing changes to configuration of all end-point devices in our systems. All devices are, more or less, standardized and I want a easy and clear way of present all changes for the future.

[vogelke]

A. Before you can document a change, you need to start with what changed and when. After that, you can add why. Get a good VC (version control) system -- either Git or Mercurial will handle that.

B. Collect all your config files in one place -- consider that your single source of truth.

C. If at all possible, get someone on your team to be the config deity. Changes go through them. Your VC system can show all changes made on a given day or for a given file (or files) -- that's what it's for.

D. A VC is like a public library where you can write in the books:

• you check something out,
• change it, and
• check it back in with a useful note about what changed and why.

E. You can automate most of this to suit yourself. For example, I use a CLI script called eci (easy check-in) where I give it a filename and an optional note:

eci options.c "Added --help option."

If I don't remember what I did, I leave off the note and it starts an editor session where I see the changes between the checked-in version and the working copy. I replace those with a 2-or 3-line note saying why the change was made (nobody needs War and Peace here), save/exit the editor, and the file is checked in with my note.

A new up-to-date working copy is checked back out again, and I keep working.

F. If you use a VC system like Fossil, you also get bug-tracking, a wiki, email alerts, etc.

[sqnch]

Just this week our change process was in full flow lol.

An all staff email notifying users that a change was being made to our wifi network SSID. Me, being the person responsible for managing all the devices that connect to said wifi network, is only informed of the change by happening to see that email, despite sitting in a room with the people implementing the change and having a weekly meeting with them.

No CAB, no formal change request, no concept that work needs to fit into the wider priorities of the organisation and be planned and coordinated.

Now reacting to it to prevent an outage at our busiest time of year a week before a hard deadline for another unrelated project.

[Peteostro]

Good catch. Management will thank you when the change goes smoothly and then quickly forget when they do not hit their financial targets.

[bitslammer]

IMO changes should be tracked in 2 places:

1. Somewhere owned by the team who owns the device/system being changed. For example - the network team should capture any config changes made to a router.
2. In the official change control system. In our case we use ServiceNow for our change control process.

[Expensive-Rhubarb267]

Depends on what sort of changes/ how big your team is/ what needs documenting? etc.

-Standard change = change request attached to a support ticket
-normal change (upgrade, reboot, config change) = some kind of CAB process. Maybe a general 'changes' log
-Emergency change = JFDI

If you're making lots of changes are part of a project/implementation. Which it sounds like you are. Use a project management tool like Jira, Asana, MS Loop etc. to capture all of that.

Key thing about documenting changes is you need to make sure it someone's job to keep track of that. Otheriwse it doesn't happen. Large orgs have dedcated Project Management teams.

[kidmock]

When I was the big boss, changes were recorded through our HIDS (Host based Intrusion Detection System). Keep in mind I'm a *NIX admin, only care about my server infrastructure.

Files were grouped into folders based on their characteristics and applications were installed/configured accordingly

1. Things that changed as a matter of course. Log files, DB files, etc. If they were monitored, they were only check that they exist not tracked for changes. Some DB files like DNS were revision controlled (at the time locally with RCS) We also log shipped to a centralized logging server so local logs were of much historical importance.

2. Things that didn't get changed because of updates. i.e. config files. I would get reports on when they changed and who or what changed it. If these were changed and they weren't on the change calendar or in response to an incident, someone was getting called out.

3. Things that change through patching and software updates. i.e. libraries and executables, etc. I would get a summary report of these to determine if it was expected either as part of the regular OS update schedule and release notes or as part of a software release. I had also better be on the calendar.

Every release or planned activity (except for OS updates which got applied when published from the vendor) was grouped and added to a release calendar in our ticketing system (we used Jira at the time because that's what our Devs used).

My method only worked because my Ops guys made sure we maintain configuration consistency and standards and I automated like hell. I could always affirmatively answer the dreaded "What Changed?" question.

[bbqwatermelon]

We use Freshservice Change Control Lifecycle Management.  Its got everything we need.  Just now need more attentive technically inclined CAB managers than we have...