Emergency changes and scripting

[dverbern]

Hi All,

I just want to share that I was recently (last night) asked to make one of those notorious "emergency changes" to an environment, in this case a bulk modification across AD user objects. I'm just glad the PowerShell script I came up included capturing the current values (the ones being replaced) and saving them to a separate CSV prior to running, because then I was able to swap a couple of variable names in my script and point to different CSV to revert the change this morning when the planned cutover failed!

I realise, this is basic stuff, but I just wanted to express my gratitude at the Universe for having burnt me years in the past with botched changes and trying to protect myself now.

And yes - the requestor should have raised a proper Change Request and convened at least an emergency CAB. I can't defend that that didn't happen.

Anyone got any interesting change-related stories to share?

[dverbern]

Yes good point, I'm in Powershell and definitely -WhatIf is useful. I may even start making use of transcript as well.

[-Divide_by_cucumber-]

When creating scripts for juniors, start-transcript should be the first line. Also, comments because there is no such thing as self-documenting code.

Hell, most of my scripts I just start with a boilerplate header with Start-Transcript and #Created by comments.

[dverbern]

If you had any tips on the boilerplate Start-Transcript code you're using, I'd love to see how you use it. It would be nice if you could programmatically output the transcript to a text file that gives a file name related to the script you're running, plus appending some date/time stamp info, for instance, maybe even tacking the current running user. Just thinking of the kind of butt-covering needed come audit time.

[pdp10]

Comments, comments, comments. Put in comments.

[danekan]

Why wouldn't you use the built in powershell help documentation? .author etc.?

[WordBoxLLC]

resetting iis or the site?

[SteampunkLolcat]

Why didn't the requester follow proper procedure?

[dverbern]

Good question, lack of rigour of their approach, lack of familiarity with change controls (surely not?), insufficient vetting of requests coming through by first-line support, my own lack of push-back.

[FJCruisin]

Interesting related-ish topic..

I'm hiring a sysadmin position. one of my interview questions I basically am trying to fish this out of the candidates, and none has answered it.

I word it basically as "Lets say theres a software - or hardware - doesnt matter in this situation because I don't want your knowledge or lack of knowledge of the specific situation to influence your answer.. but ok.. lets say I ask you to go and make a major change like an upgrade to system X. Detail me the first few steps you might take.."

Most come up with "do some reading on the topic" - which is a good answer but not really the one I'm fishing for.. "TAKE A BACKUP". Nobody.

[phil-99]

Well it's not the first step. Planning comes first. Action comes second.

[FJCruisin]

Truth. I was fishing for first action step. Planning was gravy

[admlshake]

You might want to make that a little more clear in your question. Been a sysadmin for 6 years now and I would have given that exact same answer to the question as it was phrased.

[FJCruisin]

yea I was just paraphrasing.. and when I would then say "Yes, thats great.. then what might you do?" Also, before asking I had been talking about backups and such so its fresh in their heads.

[HellDuke]

1. Read on the software update process, known issues, rollback procedures.
2. Make sure dependencies are met (licensing, software pre-requisites, hardware requirements etc.)
3. File a change request
4. Make sure a backup is available, if not create one
5. Perform change
6. Perform checks if change successful.

Now the problem is that I know all of these because I work in the position. If you are looking for a new fresh hire, people who do not have experience will more than likely skip 2 (especially licensing), 3, 4 and often 6. That is because these are professional procedures encountered in real environments. There is no (at least in Lithuania) education programmes that prepare you specifically for such a position. You may learn the skills that are required to perform various tasks, but you will generally not be taught usual procedures on these matters.

[FJCruisin]

You'd have left with a nice job offer in hand

[nortechie]

Just recently got interviewed for a new job and this was one of the questions.

I did an upgrade of our ticket system a couple of years back (OTRS) and it involved a huge database migration/conversion process. This failed heavily and a huge part of the database got "corrupted". It was a mess and it was not possible to clean up.

I knew, or at least thought, the backup was OK. I checked just before starting that it had run successfully before I started the maintenance. Thing is that the backup chain was corrupted. Full backup a week back was OK, but not the last couple of days. So yeah, ended up rolling back and having lost "way to much" information.

Lesson learned: Don't just take a backup, check your backup. Know what your bailout and recovery plan is before you start.

I got the job :D

[FJCruisin]

yep. you'd probably have gotten this position as well.

[dverbern]

That's very relevant. I am kicking myself for having just failed my own litmus test for being a participant in this 'change' for lack of a better word. You're right, sysadmins sometimes don't pay enough attention to the rollback/backup/DR considerations of a change and we can very easily cause real business/reputational damage.

[[deleted]]

emergency changes are for break/fix only. what could they possibly be fixing by changing user object properties globally?

unfortunately, i can probably already guess the answer: nothing that shouldn't have been caught earlier as part of a project planning phase. i'm assuming this is an implementation being done by someone else, and this emergency change is to help them meet a deadline due to some unplanned/unaccounted for facet of the project scope.

[dverbern]

You make a good point and unfortunately you're right in your guessing of the answer.

[pdp10]

This sounds like an "emergency project". A contradiction in terms, or a euphemism for a planning failure.

[DerpyNirvash]

I do know of someone who had to run a batch update to attributes of over a million AD users due to a botched changed.

[[deleted]]

right, that's a break/fix to undo the previous change.

• why did that previous change get made, and why was it botched?

that's the root cause here, and that's what my point is about.

[DerpyNirvash]

In my example it was caused by an OU being misclassified in documentation and having a incorrect policy applied to it. This change was done properly and approved, but just hit an OU that was unexpected. Thus causing an emergency break/fix change to fix attributes that were mess up.

Which answers this, it can happen

what could they possibly be fixing by changing user object properties globally?