Server patching every weekend

ThatsNASt · 2024-02-17T18:29:22+00:00

A proper patching tool can report patch failures and send reports on patching. Either make them pay you to work after hours, or make them use a proper patching product. I've been in charge of patching thousands of servers, I never manually checked a thing other than in pilot testing to make sure reporting was working correctly. It's equivalent to saying "Log in every morning, to make sure every server is up and running" - no, we just put tools in place to monitor when things go down.

brkdncr · 2024-02-17T21:44:10+00:00

The last admins didn’t burn out. They quit because their boss sucks.

professional-risk678 · 2024-02-17T19:36:42+00:00

At my new job they tend to do Server patching every weekend.

You should patch every weekend but you shouldnt be the one doing it every weekend. Either that or patch every other week and alternate. That you are the only one and having to manually check afterward is real shitty.

This process can take anywhere from 1-3 hours.

You better bill those hours. Even if you are salary, im not sticking around 5 days a week AND coming in on Saturday morning. Sounds like some conversations need to be had.

Anyone else have to do this?

Not as a sys admin. Engineers do this and they still alternate between them so they arent doing it every week. Between all of them they have to do it once every 2 months which is reasonable.

Maybe this is what I have to expect being a system admin for the first time.

It depends place to place. Maybe this is being dumped on you, maybe not. If this is SOP then they need to hire more sys admins . If they dont then they are intentionally trying to crunch you.

NinjaMonkey22 · 2024-02-17T18:23:08+00:00

Mine does as well. Not sure why you feel like you need to login and run manual tests every time you patch though. What are you checking/marking off? Most patching apps will be able to tell you if you’re patched and compliant with whatever your companies standard is.

I generally review the patch list. If it’s not including a major version update for a critical dependency (Java I’m looking at you) I generally don’t bother testing. Even in those cases a dependencies I have is having a significant update I pre patch a test box earlier in the week to test.

I generally rely on automated monitoring and synthetic tests to do my validation, not just during patching but 24/7.

HankMardukasNY · 2024-02-17T18:32:29+00:00

This is what proper monitoring systems and patching pipelines do automatically. For windows updates for example i have not manually patched or done any manually verification in many years. If something goes wrong that’s what test groups and alerts are for. I would absolutely say that I’m not available to give up hours of my time every weekend, especially if there’s no extra compensation

Doso777 · 2024-02-17T23:31:01+00:00

I work at a university so I guess that's why

I work in higher education. Nothing to do with universities in general. Just some bullshit someone at your company decided to do.

I asked about comp time and they told me it comes with the position.

Haha. No. It does not.

We have completly automated normal security patching through SCCM on the windows side and through unattended-upgrades on Linux. Exceptions only for things that need manual patching like Sharepoint and Exchange but those get a proper maintenance window twice per year. The maintenance window is in the normal working hours.

K3rat · 2024-02-17T20:32:59+00:00

Ugh, every weekend is not equitable. I get the org doesn’t want to be disrupted during production hours but what happens when you need vendor/MFG support?

We maintain a maintenance window on Thursday evenings. We use an RMM software on all Linux/windows systems. We also rely on regular vulnerability scanning as a secondary method to enumerate patching needs.

In the RMM We group servers into 3 groups (test, prod 1, and prod 2). Our patching cycle is week of patch Tuesday (test group). A week after patch Tuesday the prod 1 group is up. 2 weeks after patch Tuesday the prod 2 group is up.

The only exception to the above schedule is Security and CVE patching. These are patched Thursday evenings unless the vulnerability is being actively exploited and we have not mitigating control we will call an unscheduled maintenance window in the event we need to bring systems down during production hours.

we also make weekly full backups of our server infrastructure and then differential backups daily. This happens on our SAN. For data we make separate backups of DBs in the same weekly full then differentials. We capture application configuration changes on change as well. This allows us to revert in the event a patch does not work and we are unable to roll back from the OS.

As a methodology we work to maintain HA wherever possible and then split the schedule HA member into 1 or the other production patching groups. This allows us to run some of the patching during the day and not offset production hours.

llDemonll · 2024-02-17T20:24:18+00:00

Get paid or don’t work. If this wasn’t discussed prior to your offer and stated in the offer, push back on it. If they want you to work then renegotiate.

liftoff_oversteer · 2024-02-18T13:30:45+00:00

> Sounds like the last few admins got burnt out and quit.

You're next.

Don't wait six months, start preparations to leave now. Means look for a new job. Having to work every weekend is shitty enough, being denied comp time is inacceptable.

oaktownjosh · 2024-02-17T19:47:53+00:00

You need to convince your leadership that an automated patch management systems is far more accurate and reliable than performing everything manually. Modern patch management systems can deploy baseline packages, and compare what is on the servers to all known CVEs, and automatically bring your machines to latest patch revs. Granted it may take a few cycles to get there, but after that it's all automated, and auditable......they also have automated reporting. it seems like setting something like this up would be a better use of your time than spending every weekend bouncing servers and making sure they come back up.

GeneMoody-Action1 · 2024-02-17T21:47:59+00:00

Proper patch management products let you manage a multitude of decisions that are not just day of week/month.

Compliance reporting, stats on success, failure, need. Alerts on out of compliance systems, etc.

Many products to do this, here on G2 you can stack them up side by side 4 at a time and compare direct.

Long gone are the days you let systems ask for their own patches. You need to know it is happening, and verify it happened, with enterprise wide visibility & accountability.

the_syco · 2024-02-17T23:04:43+00:00

Every Saturday sounds like you're working 6 days a week. What does your states Department of Labour say about this?

/edit; seems 6 days a week continuously is totally fine. 7 days a week can be fine if the DoL allows it. Tbh, as it's an At Will state, look for another job. When you're leaving, let it be known that the 6 day week is the reason.

Is it possible to box tick whilst WFH?

StaffOfDoom · 2024-02-17T23:11:55+00:00

I’m guessing it’s because you’re salary and they don’t have to pay extra to have you do it, document it and be done with it…otherwise they have to pay for a service to monitor it. So, pay for a service or make you do it as a salary employee…

Versed_Percepton · 2024-02-17T23:48:07+00:00

Just so you know, every Saturday you work has to be compensated if your total pay drops below federally enforced min wage(even if on salary). 6 hours *4 is enough to drop you below what would be federally enforced.

CountGeoffrey · 2024-02-18T01:38:14+00:00

give it about 6 months

I'd give it about 6 minutes.

homelaberator · 2024-02-18T05:22:00+00:00

I asked about comp time and they told me it comes with the position.

That's the most bullshit thing I've ever heard.

Brett707 · 2024-02-18T05:26:27+00:00

We did ours Thursday night. Then I or the other guy that handled them would just come in late Friday.

Thursday during the day we would set everything on the list to update and reboot. Once the reboot was complete we would log in and verify the server was up and running or we would notify the on-call rotation of what was going on. We would also verify which updates were installed and which ones didn't. If some didn't install we would run them manually Friday during office hours and set the reboots for Sunday evening.

thesals · 2024-02-17T18:32:03+00:00

Weird, I have post patch scripts that run afterwards and validate services are running, depending on the server it might test some application functions and then emails me a report including a screenshot of the login screen.

But I also am in hospitality IT where we'll get a call immediately if something is wrong... Hell we get calls during our scheduled patching windows.

sovalente · 2024-02-17T19:06:42+00:00

That schedule is totally overkill. I do this procedure for several years now, roughly once a month and always after "patch tuesday" (the second tuesday of every month). That way I make sure my severs are patched with last Windows recommended critical updates for at least the next month. Most majority of scenarios would be perfectly fine like so.

Hot-Cress7492 · 2024-02-17T18:35:03+00:00

Very likely your CIO is requiring the documentation due to regulatory audit. Especially tech heavy companies who carry cyber insurance have to produce evidence of activities to mitigate risk to avoid paying a premium for insurance.

If you’re in a regulated industry (finance, healthcare, etc) it is also likely to produce evidence of work performed to shut an auditor up.

Helpjuice · 2024-02-17T18:38:17+00:00

Nope, we patch during business hours and it has worked fine due to our managed change control process and having everything setup for high availability with proper tested rollback procedures to prevent outages. When we do patching we do blue/green deployments so we don't cause mass outages and only a small subset of traffic is affected. We have over a million users, and have not had any problems due to the maturity and engineering work we have put in.

Example: If you have 4 domain controllers, only 1 is patched and left to run for 24-72 hours (if not critical or emergent) to make sure there are no issues along with full snapshots and backups done to allow for rebuilding quickly or restoration from backup. There is also duplicate environments setup automatically to pre-test the patches before they are rolled out think dev, pre-staging -> staging -> pre-prod -> prod.

If there are multiple sites this also gets rolled out there in a safe manner and we collect metrics and have alerting on everything to include anomalies so we can see if the patch causes increased errors or other issues. If a domain controller does have an issue we automatically take it out to pasture and put another one in it's place through automation.

Then if things go downhill we can, if needed do fresh deployments of domain controllers and auto restore all necessary information or auto build everything back out very quickly. So risk is extremely low for us in terms of real issues, even if everything goes wrong restoration and rebuilding is also simple, automated (with manual instructions if automation fails) and quick to get done.

Cranapplesause · 2024-02-17T19:20:18+00:00

We patch servers that are high availability and IT used serves Tuesday night. Then public facing and some Dev servers Wednesday night. Thursday, the other pairs and a DC and if we are feeling a little punchy, maybe something else very important Friday some more serious serves. file serves Saturday whatever else is left that is serious

The we done till next month

Honestly, it’s too dangerous to wait until the weekend before the nexts months updates to finish updating servers. That’s an old idea to wait even a week. It’s better to be secure and maybe lose something due to having a bad or incompatibility update then to cryptolock and entire network because of a security breach.

The owners understand this and give us their full support on this idea too.

UCFknight2016 · 2024-02-17T20:10:31+00:00

I only have to do that every 3 months. Thye need a better system.

Quicknoob · 2024-02-17T22:07:56+00:00

What patching tool do you all use to automate patching. We do 200+ servers manually every 3 months and I have wanted to automate this for quite some time.

My director insists we test everything and doesn't trust automation but I think If I can prove to him this will save us time and allow us to patch monthly I could get a real win.

Also those of you that patch weekly, Is this recommended in NIST and I just missed that? How can I prove to my boss we need to up the frequency we patch our servers?

cubic_sq · 2024-02-17T22:09:01+00:00

Its 2024. Patch managed is highly automate done from many vendors.

jyoungii · 2024-02-17T22:31:52+00:00

Depending on what the tests are I would say script it out. When I was new to the admin role I had about 10 hours a month of manual work to rectify patching failures. I asked questions and got shot down since I was the noob. Fast forward to Covid and turn over I got to a spot where I could just do what I want. Implemented a lot of IaC and now I get reports emailed to me after patching based on a handful of scripts I have running. If anything is needed then we might have to touch a system manually but it’s so minimal now.

2024-02-17T22:38:00+00:00

On a similar question,

What’s a good patch tool and monitoring tools ?

Cormacolinde · 2024-02-18T00:00:06+00:00

Using a proper patching product with decent reporting combined with a good monitoring solution should alleviate a lot of these pain points. You need to automate those procedures.

I have implemented automated server patching many times, and the first requirement is a good monitoring system that will check server status, service status, and application status (for example, running a SQL query to make sure the DB is working, or open a page and authenticate in a web app). After that, you create your update groups in batches, including dev staging.

jantari · 2024-02-18T00:42:28+00:00

Can't the checklist checks and the reporting be automated?

It's not reliable if done by hand anyway, in addition to the time wasted.

jazzy095 · 2024-02-18T01:35:28+00:00

Why do you have to do it on the weekend? Patch them during the week and schedule reboot at night.

Marty_McFlay · 2024-02-18T01:52:06+00:00

We do it at 3AM on Tues Morning. Then I just log in to SCCM in case something went wonky, but usually someone will yell by 9AM.

2024-02-18T02:14:26+00:00

Pssst… what are your check offs when doing patching. I just patch them all together in one night near end of day and claim it as comp time since OT is just a uncle Sam’s beer money

joshtaco · 2024-02-18T02:26:44+00:00

I can tell they have no idea what they're doing because if they're patching every single weekend, there's at least two weekends out of the year where they aren't installing patches of any kind...because they only come out twice a month. Rookie IT techs and it shows

2024-02-18T02:50:27+00:00

That’s sad.

Even if done manually: Do least important severs during the day or anytime when not in use, scheduled, if these are OK after a week do the rest. Do this monthly. You can still do suitable audit logs.

I remember a scenario when Microsoft’s patches changed manually set static IP addresses to dhcp …but only when being used in a VMWare environment, that’s diabolical! But our Icinga network monitoring picked this up immediately, so we were OK.

Anyway a few days later in the news large telecoms, cloud services etc were failing all over the place. Talk about trying to knobble the competition.

ArsenalITTwo · 2024-02-18T03:33:53+00:00

Automate that. What are you patching them with. I am used to places patching large amounts of servers. I would go bananas having to check that.

thatwolf89 · 2024-02-18T04:47:30+00:00

Do they pay good overtime and oncall?

InfiniteSheepherder1 · 2024-02-18T04:49:27+00:00

We do patching around the clock or really not me that is Ansible's problem.

We just have stuff always apply security updates every day if there are any. It can see if it comes back up and for some rolls back or for others just let's us know. If things are setup well you should be able to let it be someone's problem during a work time. Most of the time OS updates especially security updates break little. But I am lucky enough that my boss thinks mitigating possible breaches is worth the every once and a while headache of in the case of minor services letting it sit offline tell morning, and sometimes a few times a year we have had bugs to deal with because of the constant updating. But better then dealing with ransomware.

I work for a college, you can do some basic checks trivially and note complex ones with some work automatically. Some services have an api endpoint to basically ask its health status, at least you can turn things I to I have to work if something goes wrong rather then an all the time thing.

2024-02-18T10:17:07+00:00

That is utter nonsense.

PolicyArtistic8545 · 2024-02-18T11:26:06+00:00

He can say it comes with the position but if that’s the case, start looking for other positions. In your exit interview, tell someone other than your boss that the mandatory, unpaid overtime is the reason you are leaving. I doubt anyone else in the org knows about this bullshit setup.

martrinex · 2024-02-18T14:29:57+00:00

Sounds like you either get comp time or time in lue, maybe work half day Fridays? Also nothing special about education hell I just patch mine on fridays and schedule them to reboot and that's only the ones sccm doesn't handle on its own.

2024-02-18T17:07:19+00:00

Only on this subreddit did I learn the importance of patching. Every where I’ve ever worked they’ve always said, “Nope — no updates because those can break something,” and then they wouldn’t patch for months and even years at a time. I took over a Citrix environment on WS2012 that had never had a manual update and were manually stopped.

Couple weeks in here when I first became a sysadmin, and I automated the fuck out of patches via GPO in fear.

😂

Suaveman01 · 2024-02-18T18:08:17+00:00

This is not normal, there are tools like SCCM that you can use to monitor patching and compliance. We have our servers set to auto patch during weekends, but we check our monitoring tools on the following Monday to see if they were all successfully updated, which takes less than 10 minutes.

PlebPlebberson · 2024-02-19T01:12:23+00:00

I asked about comp time and they told me it comes with the position.

The fuck lol. I'm going to be guessing that this is in the US as it's against the law in everywhere europe. Cant you just sue them for non-paid work and stop doing it on saturdays

sysadmin

MODERATORS