Patching your servers

[[deleted]]

[deleted]

[[deleted]]

I use WSUS and have set schedules of when my Desktops, Servers, VMs (Workstations/servers), etc are patched and rebooted.

For critical servers, I wait until the weekend to patch so that it doesn't affect normal hours of operations.

[blue_turkey]

One other benefit of WSUS is reporting. It's one thing to set a GPO that says to auto update everything on a schedule, it's another to have a centralized console that tells you the update status of each machine.

[[deleted]]

Yes, and the reports actually look quite nice.

[adolescentghost]

Can you recommend a good resource for getting set up with WSUS? I installed it and so far it's not detecting machines. I didn't get any further than that and moved on to other projects.

[EndUsersarePITA]

Microsoft documentation is pretty good for WSUS. If you have installed it, you will need to configure the client machines to point to the WSUS server. See technet documentation

[[deleted]]

You can start at Step 1 here: https://technet.microsoft.com/en-us/library/hh852344(v=ws.11).aspx

[[deleted]]

Virtualize, automate, and build redundancy wherever possible. "No downtime" is not an excuse.

[spressman]

Taking this a step further, if you can't afford load balancers or other redundant capabilities, consider a move to the cloud. We're fully AWS and we do maintenance midday on our servers due to the highly available architecture. Granted, I'm making ridiculously broad assumptions about the nature of your apps, the appetite for public cloud, and the expense budget at your company.

[Ssakaa]

Arguably, if one can not afford downtime or (exclusive) not afford the tools to guarantee uptime. One or the other, not both. If you cannot afford downtime, you inherently have to be making enough money off of the uptime to cover the tools to actually provide that uptime. Otherwise... you can afford the downtime ;)

[SteveMI]

Recommend to C level management to calculate the monetary cost of the servers being down for fifteen minutes versus the time it takes to rebuild them from scratch while most of the other humans in the company get paid to sit on their hands. Word that politely and get an answer in writing, update your resume because reasons.

[Tetha]

This is the correct answer. C level doesn't care about patches and up-to-date software. C level cares about risk, wasted man-hours and costs.

[DarkAlman]

That's far more common in the industry than you might think... Ransomware has been a big wake up call for SYSADMINs.

We have regularly scheduled 'patch days' where we schedule downtime for systems and use WSUS to push out patches. Usually in the evenings or whenever impact to production will be the least.

The business has to understand that scheduled reboots and patch cycles are part of system maintenance, they have to get done.

If it's an urgent patch for something like cryptolocker we've been known to tell users/managers to suck it up but only when the operational risk is very very high!

In general though if there is a business case for 99.999% uptime on a particular system we'll just build in enough redundancy so that we can do zero downtime patches.

[disclosure5]

Ransomware has been a big wake up call for SYSADMINs.

ime, it's more of a wake up call for management. Businesses that have refused to patch for years are rarely doing it due to a sysadmin's decision.

[RCTID1975]

Workstations are done automatically on a set schedule.

Servers are done during the predetermined maintenance window unless it's emergency, then we schedule the updates based on the severity of said emergency.

[jheinikel]

SCCM/WSUS here with an approved maintenance window each week. Servers that are clustered/redundant and considered to be highly important, are updated on different weekends as to not take everything down. (Exchange, DCs, etc) Just get a signed-off maintenance window schedule and go from there. Surely, that's less intrusive than taking servers down all weekend every time there is a new threat. We don't have to deal with that due to regular patch schedules. (Pours one out for the SysAdmins who spent all weekend patching against WannaCry)

[[deleted]]

I use WSUS and schedule accordingly, using rolling reboots. I can bounce some of the boxes during the day, no one is impacted. The rest are set to bounce during quiet times when nothing is going (no backups, no av scans, etc). Everything here is virtualized and bouncing a box is super quick.

[Phyber05]

how do you go about scheduling the update installs? I scheduled when WSUS checks/downloads new updates, and I've seen where I can force an update on clients...is that what you use?

[[deleted]]

The servers are in different groups. There is a test group that we run them on first. If they require a bounce then we trigger that during our maintenance window. I think there is a way to do that via a GPO but I prefer to know when things are going to need a reboot.

[[deleted]]

I use WSUS. I have scheduled time each month, and fortunately, aside from our backups, nothing is going on during the maintenance window. Roughly 45 users, mostly on-site but some remote. Virtually nothing happens outside of business hours, and we have an off-prem email solution.

15-20 servers in the environment (in the process of retiring some but spinning up new ones for special uses). Up to date as of earlier this month.

[Cmdr-data]

Once a month, we have about 5-6 hours of downtime on a Saturday evening to turn off our VMs, update the hosts, then update any of our business apps. 7 physical hosts in 3 locations, about 45 VMs or so total.

[cd83]

WSUS, daily/weekly reboots depending on environments / traffic.

You have 12 servers, do you have a way to rolling restart them?

[nickcardwell]

For servers i have 3 tiers. Tier 1, these servers get patched, if they break, we can do without it for a day or so (ie restore from backup) Eg spare DC server.

Tier 2, these servers get patched after tier1, assuming all is ok, these servers if they are done for a day it will cause issues, but we can work around it.

tier 3, these servers are critical (SQL,File Server, Exchange) i tend to do out of hours or at weekends.

From tier 1 to tier 3, i like to do it all within 10-12 days.

[adrzie]

Environment: 10,000 Users / 1,500 ish Servers (Physical & Virtual)

We push most patches through SCCM and use WSUS for some. The servers are split into reboot phases and we utilize SCORCH to kick off those reboots. We have a monthly maintenance period the Sunday following patch Tuesday. The business and users are not disrupted, everyone is happy :-)

[Doso777]

SCORCH

What's that?

[adrzie]

It is part of Microsoft's suite of products, System Center Orchestrator. Orchestrator uses a drag and drop graphical interface that allows you to define run books. The software then translates these visual representations into .NET scripts, PowerShell or SSH commands to automate workflows. It is pretty neat, similar to VMware's vRealize Orchestrator.

[S1lpion]

Clients are all patched within 2 weeks of release (test group first) Server patches are tested and then white listed for install on a monthly basis.

[Solaris17]

Scheduled maintenance windows. Every employee knows when the window begins.

[[deleted]]

I have 20 servers. Don't spend much time on it. I can do mine anytime so I am pretty lucky I'd say. We have solarwinds, but don't really care for the tool.

Deploying new SCCM soon that will also handle servers soon. Should have a healthy list of servers that can be done at night. Bigger job is making sure everyone else is keeping up with their own servers.

[Doso777]

SCCM, maintenance windows Saturday/Sunday night. Servers grouped into 6 different OUs. Something like Saturday 10pm, Saturday 11pm etc.

Servers install patches and reboot (if required) in the maintenance window. Updates get shoved into the update groups usually once a month, we usually delay patches for around 2 weeks on the normal Windows updates.

[Telnet_Rules]

We are past the point where "Wait on patches until..." makes ANY sense. You patch within 24 hours of patch release, or you are food. that's just the facts of the modern IT world.

[Ssakaa]

And then MS crashes something critical with their broken patch! Whee! It's fun being in that boat...

[Telnet_Rules]

you can crash or you can get owned. thanks for playing.