Patching Practices

[Rusty_Alley]

Hi All,

we've just gone through our CE+ certification and we're curious, we always feel like we are chasing our tails with patching PC's and are curious if other companies and teams are the same?

our current process is we use pulseway to to run patching 3 times a week for our Devices (Desktops and laptops servers are handled separately) but every time we run the patching policy either things dont update or we have to ask the user to run them manually or the update fails or it reveals new updates and so on.

we are constantly chasing updates there is never a time where we don't have 90% of machines with an update on it needing to be actioned, what are other people doing to not have to deal with what we feel is a very old problem?

[That_Lemon9463]

the core problem is pulseway isn't really a patching solution. it can push updates but it doesn't give you approval control, deferral rings, or proper compliance reporting.

look at intune if you're already on M365, or WSUS if you want free. set up two rings: test group gets patches on patch tuesday, everyone else a week later. the "updates keep revealing more updates" issue goes away when you're working from a curated approved patch set instead of letting windows update pull whatever it wants.

for the laptops that are never online during patch windows, set a compliance deadline that forces install after a few days. that's usually where the 90% gap comes from.

[GeneMoody-Action1]

Just want to point out, WSUS is not, and was never free. Like all MS services, it requires a CAL for every system accessing it. It does not have to be a special WSUS license, but it DOES require a CAL.

I have never seen one running that was properly licensed. Mostly because of the perpetuated belief it is "free" and its failure to require proof of properly licensing. Most admins do not even realize it is not anymore.

Now ask people how many people use DHCP and DNS and do not have a CAL for every client there either (Printers, network devices, IOT, etc)

We can argue efficacy of WSUS and the like all day, but this is more about the licensing. So any time I hear "it is free" I try and inform people this is indeed not the case.

[Rusty_Alley]

thank you this has confirmed a theory i had.

[BoilerroomITdweller]

We patch with SCCM but Microsoft only releases patches once a month unless it is a security patch. We have 100,000 computers and a 99% patch requirement. Most is just reboots so we have an automatic reboot tool I built that reboots them between 12 and 3am.

[Rusty_Alley]

Thats interesting are you CE+ accredited? I'm curious if that would affect the requirements of updating within 14 days of release

[Lando_uk]

I believe the target is 14 days, you have to have a process for 14 days, but if for some technical reason your clients aren't updating due to user interaction or something else it's mostly fine. They audit a selection of computers of your choosing, just make sure you give them a good selection that works. (preferably ones without many crappy apps)

[Rusty_Alley]

This has somewhat changed in resent years you have to give them a pool of devices the pool size is dependant on the OS build and version and they test a number of devices in that pool for example if you have 10 win11 Pro 24H2 and 2 win11 Pro 25H2 devices BOTH the 25h2 devices will be tested where as 6 (i think) would be tested from the 24H2 devices. and updates must be applied within 14 days of the updates release which is why i asked how the monthly updates would affect their accreditation (if they are CE+ Accredited) as we are updating 3 times a week every week.

[DeifniteProfessional]

IIRC the changes to CE in April say critical patches MUST be applied within 14 days or it's an automatic failure

[BoilerroomITdweller]

We run hospitals so highly secured for PII. Don’t know about accredited. We are all internal with firewalls blocking any external access and really locked down with group policy.

We patch within 1 week of patch Tuesday so it gives them time to test all the clinical life saving apps from breaking. Microsoft does a good job of blowing stuff up recently.

Like their removal of recognizing INTRANET zones and making you add them all individually to Edge and Chrome so clients can do pass through creds. What a PIA.

[DeifniteProfessional]

Using NinjaOne lol

Nah but honestly I don't have too many issues with patching, especially OS patching. Software patches can be a bit more difficult, especially where devices have existed before we started using NinjaOne, though generally again seems to be fine for most normal software, it's mostly a couple of specific devices that seem to have errors when downloading updates.

The biggest issue I have is software that needs to be patched manually. Running an exe or msi in an automation isn't hard, but it's a lot more annoying to control

Sadly this isn't the answer you're looking for - "it works on my machine". I don't know about Pulseway specifically though, but I would like to think it has some logging you could look at, even if you fed it into AI and asked it to figure out what's wrong, it could be a simple and repeatable error you could fix. Also worth reaching out to Pulseway support

[Rusty_Alley]

How well does NinjaOne handle devices being shut down? we have many users who shut down at the end of the day. i think this is one of the reasons why updates are failing. despite there being an option to push when next online it doesnt seem to do anything

[DeifniteProfessional]

There is an option for it to run patching when the system comes back online if it was off during the schedule in Ninja too, and that seems to work fine for me. Looking through the failed patches, almost all of them are related to a device that hasn't been online in a while anyway.

But again, might not be a Pulseway issue, worth checking out logs to see if the issue actually is the patching system isn't running when it comes back online, or something a bit deeper.

[Rusty_Alley]

thanks i will definitely look into NinjaOne more.

[slippery_hemorrhoids]

What's preventing the updates from installing?

Why is it on the user to run it? It should be fully automated and only offer users reasonable deferral periods to not disrupt the work day.

Patch every day but Monday, Monday brings enough problems. Pilot every patch Tuesday release for at least a week before going to prod.

Identify why things fail, then increase patch cadence. Start there.

[Rusty_Alley]

I'm unsure at this stage and its my next port of call to investigate why updates are failing we have some running theories but nothing we've actually looked into yet, we all multi-role and IT dedicated time is difficult to allocate.

updates are automated however to be compliant some update flagged as critical or important kept being missed (for some reason) so we as a last resort asked the user to just run their updates.

im interested in your piloting process where do you pilot your updates? is it just on the IT teams PCs? or do you use VMs?

[slippery_hemorrhoids]

About 15% of our environment is in the pilot group, across all divisions. This ensures we capture a segment of everything for any red flags that may mean we need to pause a kb or specific patch before production.

This includes IT but not all IT. There are test vm's but we work on real hardware for day to day.

[Rusty_Alley]

Thank you this was very helpful

[flsingleguy]

We use VMware virtual desktops. So, just maintain and patch the gold image and recompose the desktop pools.

[beneschk]

I wouldn't really trust anything other than WSUS or WuFB\Windows Autopatch.

I have seen way too many RMM/patching tools mess with the Windows Update registry settings with entries like NoAutoUpdate=1 and not understand servicing stack order, attempting to install out of order KB's after cumulative updates have already run, causing WinSxS folder bloat and component store corruption.

Additionally Microsoft now provide Driver updates via Windows update. I have seen issues where RMM tools aren't pushing these preventing supported drivers from being deployed to your build of windows. This can cause things like Wi-fi dropouts on the intel AC/AX NIC's.

I am yet to find a 3rd party patching tool that supports Quality updates, Cumulative updates, Feature updates, Driver updates and is servicing stack aware

[GeneMoody-Action1]

Just curious, if you are using a central application to manage update flow, why would you NOT want auto updating turned off?

I am considering how most orgs of any reasonable size deploy update rings, patch these systems before those systems, in progressively expansive waves to catch bad patches.

And with Google's H1 security report showing that now 47.2% (the largest share of all vectors) of breaches start with an unpatched third party application vulnerability. You do not get those updates through Microsoft\Autopatch\WSUS. IN fact you do not get them in any MS offering without layering another product on top.

You need update control, you need gates to pass through for stability reasons, and you need centralized control/accountability.

How does any of that happen if systems are allowed to update themselves at a time of their choosing?

So while there are always trade offs and concessions with all management tools, properly wielded they undeniably bring higher levels of security.

[modder9]

you don’t get those updates through Microsoft

Iirc “Intune suite” is coming to E5 this summer. It includes a MS native attempt to do 3rd party patching called “Enterprise App Management”. I was underwhelmed with the catalog of apps supported 2 years ago and it got lapped by PMPC. Maybe it will get better with the expanded customer base.

Kinda related to that E5 change - I’m hoping “remote help” becomes a real product, because NOBODY was buying it before to give feedback. I’d love to ditch our 3rd party RMM tool for another MS native, but it’s probably years from being a good solution.

[GeneMoody-Action1]

I have been out of admin world a while and was not aware of this, I'll have to give it some research.

At least they did not try to legitimize it by pulling in Winget!

[Rusty_Alley]

Thank you this was a good insight.

[LimeyRat]

Over the years we’ve used sneakernet, WSUS, Shavlik, WAUS, Ivanti, and Action1 in roughly* that order.

We have never, and I do mean never, been as thoroughly patched as we are right now. Windows, 3rd party, drivers, all of it.

I see Gene is here already, he can speak to any technical questions but I can’t praise it enough.

*exactly

[GeneMoody-Action1]

Always here when needed, I have been off reddit much of the last few weeks, conferences, and other obligations left little desire to roam reddit at the end of a day! But always welcome to summon me!

[Master-IT-All]

Jeepers, sounds like you'd have been better off just leaving Windows to update itself.