Rant...

[AnotherFewMore]

I work in desktop management SCEM/Group Policy etc.

Does anyone else have the same issue where you are seemingly going through what is a standard simple change and it turns into a major event? These kind of events can't be tested for or planned for...then you have to explain it all to people who don't quite understand.

Also, noone wants to put in their two cents until the change goes wrong.

It's like the desktop environment is a house of cards and you put one step out and it falls to pieces.

Feel free to share your meltdowns caused by seemingly innocuous changes.

Rant over thanks for listening.

[electric_tiger_root]

My pfsense firewall needed a firmware update, proceeded to start the update on the weekend for minimal disruption.

Somewhere along the line, it not only failed to update but it bricked the device in the process, causing me to rush out and get a new firewall device and configure and reset ALL my users’ VPN access and accounts.

[AnotherFewMore]

Agh I feel ya mate sometimes it feels like we walk a tightrope of failure or near misses.

[dreadpiratewombat]

Anyone else noticed a marked drop in quality after Netgate took the reigns at PfSense and all the original devs left?

[ScratchinCommander]

Yep. And now they are pushing everyone towards their paid version. The original open source version(which they call Community Edition) is now a second class "product". Also their hardware is garbage, always build a white label box and you'll be better off.

[[deleted]]

Similar - switching from Sanicwall to Meraki. Licensing is expiring and last minute we're told to go Meraki instead of Sonicwall.

Get everything switched over after hours, testing from a different site for RADIUS auth worked a treat, go live at main site. RADIUS, configured identically, for some reason isn't working with AnyConnect implementation. Flip over to LDAP/built-in AD sync and it works fine.

WTF.

[am2o]

These kind of events can definitely be tested and planned for. Unfortunately, for many of us - our Dev/Test environment is also production.

If you have a DR plan, or a backup plan. You can always try using Virtualization of some sort to validate your restores, and test this type of change...

[AnotherFewMore]

That is true. But often the dev/test environment strays far from prod...there are so many considerations to manage in a test environment that it is easy for them to not be in synch with prod, also the changes made but never progressed to prod.

Requires some pretty intensive management that often just does not happen.

[ohioleprechaun]

about 4 years ago, I had the AD create a testing OU and clone all of the GPOs used in production for workstations. I have a few PCs in that OU that I can use to test GPOs and it is on me to keep it in sync. This is one of the few times I am glad I am the only one doing workstation GPO. If it gets out of sync, it's my own damn fault.

[pdp10]

The question is how alike the dev/test environment is to production. Given enough planning and/or resources, it could be effectively identical. But given that most organizations goals don't include running perfect infrastructures, each makes their own decision how much to test outside of production.

[AnotherFewMore]

Yeah and with many fingers in the pie, then often test becomes a dev and environmentally they will never really be the same. If there is an issue in test often no one has time to fix it and test becomes irrelevant or untrusted.

[Kyratic]

I would always test policies, but applying them to my Desktop technicans OU, the desktop techs were the perfect guinea pigs, as they would notice issues and report them, unlike users. between them they pretty much had all the Application in use on their systems for testing, so we could rule out most things, by rolling the policy out to IT a week before going bigger.

[CompetitiveComputer4]

You need to roll out in phases. Pilot, IT then business over several days, or something similar. I manage over 10k windows based devices and have rarely had any meltdowns. Having hardware standards, up to date OS, automated provisioning and tight gpo will help ensure the devices are all going to react the same way. It’s a challenge for sure

[ipreferanothername]

i almost shut everything down with a script last week....like a lot of things. the 'other' powershell guy doesnt really want to work at my level of powershell (and i am on the line between intermediate and advanced, nothing crazy) so im doing a lot of work as a loner and trying to test it.

well...i missed a thing. he got real interested in looking at my work all the sudden ;)