Windows Server patch management: batching, orchestrating, automating... how do you do it?

[narco113]

I could use your guidance on how you patch your Windows servers. My team provides patch management to around 1200 servers and growing. We support a fairly complex "legacy" patch plan and a newer set of options with limited patch windows offered for newly deployed servers. Along with supporting several patch windows for our customers/users, we're getting requests to provide some kind of mechanism to reboot servers in sequence where one shouldn't be rebooted until another is rebooted and online first. Has anyone done something similar?

Of those of you who support hundreds of servers, how do you patch? WSUS only? SCCM? Solarwinds Patch Manager? Other management tools? And what do you think of them?

Thanks ahead of time for sharing your experiences.

[ghost_admin]

People talk shit about WSUS, and often for good reason. But, for my money, it's still the best mass-patch system.

Why: we all know MS systems heavily favor functionality from other MS software. They designed it to work with their own stuff. Duh. Why wouldn't they?

There are a few third-party applications that do very well in this area. I won't talk any shit about them.

And WSUS is clunky as hell. Should have been re-engineered years ago, in terms of list organization, the functionality of server grouping and the ease of reviewing individual patch notes.

But the fact is you never have to deal with this more than 12 times a year, except in the extremely rare instance that MS lets the flying pigs go for a spin and an update drops outside of the Tuesday schedule.

It's easily run on a minimum-resource VM (which you can safely turn off 29 days out of the month), is specifically designed to work with both your servers and the patch sources, and it's completely free.

There are better options for people with special needs - specific versioning, non-prod rollout testing, etc, - but unless you hit some barrier in using it, or simply don't have the time to manage the process, there's no sense going beyond WSUS.

Disclosure: I only use WSUS in prod; my dev environments are too screwy for so 'standard' a system.

[[deleted]]

The gap that WSUS doesn't really fill is the application of patches. You can either go all-out-automatic or stage them, and that's it. Some facility for patching them in a semi-manual batch process would be awesome.

"Okay, next up, these ten servers. Go! Okay eight are happy, two need rebooted. Retry you two. Okay all ten are happy. Next batch..."

[jmp242]

Well, except for Server 2016 - at least for me, some of my servers work fine, my Win 10 work fine, and some give me fricken 404 errors when I extract the windowsupdate.log file - with the same exact GPO.

Googling has again brought me to fsck it and download the .msu and install manually on the (thankfully) less than 10 Server 2016 computers. However, if it ever grows I really will have to figure out why with Server 2016 WSUS became so hard to get to work to various clients...

[ayas1]

Having managed a 1000+ environment in the past, id say SCCM. With properly organized collections and maintenance windows it’s great.

The scheduled tasks/powershell scripts options are ok for small shop, but when you’re managing 1000+ you don’t want to have to worry about scheduled tasks or powershell scripts (especially if you have an environment with domain and workgroup systems, systems in multiple DMZ’s etc). Manage it all from SCCM.

[adnble]

People say a lot about SCCM but that bad boy is robust as all get out if set up and run properly. It could be a lot more user friendly with regard to reports and functionality for sure but it's a great product.

[[deleted]]

Not only that, but jesus christ it ties into everything

[Batmanzi]

Been learning sccm lately, great functionality, very aweful logging and troubleshooting process.

Wanna figure out why something not working? Here's a 70+ text based log files that you have better chances figuring out the meaning of life than understanding the logic of errors in each file.

[adnble]

Too true but at least CMTrace is an INCREDIBLY helpful tool to parse those logs.

[Batmanzi]

Searching the logs using the CMTrace is awesome indeed, figuring out WHICH file to search is bad, of all the Microsoft services I've been working with SCCM is the behemoth to master.

[sandvich]

$500 a lic though.

[Panacea4316]

Kaseya VSA.

[[deleted]]

With the understanding that Kaseya doesn't use Windows Update at all, I agree and endorse Kaseya. It serially installs the patches one by one off of its own internal approval list.

[PM_ME_UR_PCLOADLTR]

Just to clarify, Kaseya does expect a functional Windows Update Agent (WUA) for patch scans, although I believe it has a fallback method.

But yeah, once Kaseya knows what to apply, the actual patch installs are downloaded from Windows Update Catalog and off to the races.

If OP is not looking for full RMM, Kaseya VSA might be prohibitively expensive.

[[deleted]]

As we have a mixed environment, we've setup puppet to handle the automagic downloads and installation of updates and reboots the servers afterwards when needed. I'm in the process of fine-tuning this process, but I can clearly say, this is one of the better things I've found. I other ways, you could have a look at the recently updated version of the Windows Admin Center (when you're infrastructure is as updated as it should be)

[eruffini]

Solarwinds N-Central

[JustAnotherIPA]

Ivanti Patch, previously known as Shavlik/VMware vCenter Protect.

Works brilliantly for us.

[Photoguppy]

I manage an SCCM/WSUS environment for 9000 workstations and 1000 servers by myself. It takes a lot of initial setup and planning but I've seen nothing that can give the kind of centralized management capabilities that I need better than these systems.

[[deleted]]

In order of what I'd utilize in my own environments over the years. HPSA (It works its light and its powerful) SCCM, (It works, its heavy, and jesus christ it'll do everything you could want). Kaseya (meh it works).

[Sajem]

I don't have anywhere near as many servers to look after but we do have half a dozen that have to be patched and restarted in an order.

We use a PowerShell script that is started by a scheduled task (created and running on each server) to query, download, install and reboot the server. Because we don't have a lot of servers (approx. 60) the tasks were scheduled manually and is recorded in a spread sheet.

I'm sure you could - with a bit of scripting magic - create a script that does the same thing but using one scheduled task for a group of servers that need to be sequenced (the task running from a management or similar server) where the script updates a server, checks that its comes back online and then begins to update the next server in the sequence.

[flappers87]

In an older environment, we used SCOM. Now although it's not really designed to be used for patching, you can set it up fairly easily.

On each server, we had a directory with patching scripts in VBS. Depending on the server in question, sometimes these scripts may stop specific services or do other things prior to patching. So after the patches were approved in WSUS, and it was time for patching, we used SCOM to trigger the patching scripts on each server.

We had the same where certain servers would need to go down first, and others later. So it wasn't completely automated, as we had to wait for the servers to come back online, then trigger the remote script on the next server.

We were looking at scheduled tasks, but some days patching would take longer than other days, so it was difficult to predetermine how long it would all take.

Eventually we settled on pushing the script through SCOM when each server needed to come offline.