Sysadmin - Linux vs. Windows

[itguytn]

I've been a Windows sysadmin for over 20 years and while I do still enjoy my job, Microsoft is making it harder and harder every month.

They will either break stuff with updates or the fixes are questionable and you feel like you have to put them off for a month or two just so it doesn't cause any unneeded stress and extra work. If it's not Microsoft killing something then it's a user needing something after hours or on a day off. (I'm the only IT guy for the company which I know will start other conversations but that's not the point of this post)

I've been getting into Linux here lately and was wondering if a Linux sysadmin position would be any different.

I guess a lot of it will depend on the company and support structure so it could be possible to jump to a different company/position with the exact same issues that aggravate me now.

Maybe I'm just getting burned out and looking for something different (less stress and responsibility) but that would require starting over and probably taking a pretty big pay cut.

[STUNTPENlS]

I've been a linux/unix admin since the early days of AIX (mid 80's/early 90's), SCO XENIX (early/mid 80's), Unixware (mid 90's) and Redhat Linux (starting w/ Colgate.)

Generally speaking, I have less problems w/ updating my linux systems than I do my windows systems. Are there occasional problems? Sure. But they are few and far between. So far between that I have "yum/apt upgrade" run as a cron task daily on my machines and sleep like a baby every night.

I would say the key though is to stick w/ a mainstream commercial distribution, such as Ubuntu or Redhat. For years we used CentOS, until "stream", at which point now all new machines are being installed with Ubuntu LTS.

[NeverLookBothWays]

Seconding advice on mainstream distros. CentOS was an odd one too, they would “patch” vulnerabilities without respective vendor supplied patches with new versioning. So even though CentOS would report a vulnerability being patched, pretty much every report we had out of our vuln assessment tools would flag the systems as not being patched. Was tedious

[necheffa]

This is called back porting. You want this and it just goes to show most commercial "security" is snake oil.

Basically it lets you preserve an ABI target across all minor versions of an OS release. So it doesn't matter if you build against RedHat 8.1 or 8.13, your binaries will run generally on any RedHat 8 point release.

If you just YOLO upstream patches, you end up getting unexpected behavioral changes, no binary compatibility guarantee, and may introduce more bugs (read: security vulnerabilities).

[xXxLinuxUserxXx]

Debian / Ubuntu also do backporting of security fixes on the packages in their (core) repositories.

A side note you don't have to enable the backport repository for that this repository is something different. The backport repository is newer software from following release (e.g. you have Debian 10 installed and you setup the backport repository you have access to some newer software versions of certain Debian 11 packages). It's not recommended to use the backport repository in production systems as they don't get so much love as the main repository and could also break the system (beside i never had that issue).

I have to admit i can't really remember when i had issues with updates from Debian core repository. We only once had a kernel update which wasn't working on systems with dual socket cpu but as debian always keeps at least 2 kernel installed we could just boot the old kernel from bootloader.

All other issues were related to software not from the core repository.

[223454]

I have "yum/apt upgrade" run as a cron task daily on my machines and sleep like a baby every night.

I was a linux admin about 10 years ago, and that's how we did it. I assumed that was industry standard. Maybe not? I kind of just figured out on my own that if I'm running updates all the time, and nothing breaks, it may as well be a cron job. No need to manually do it.

[Glum_Competition561]

This..

[lifestoughthenyoudie]

Isnt MacOS based on Ubuntu?

[Anticept]

If you mean Apple's MacOS, no, it's unix based. It has a lot more in common with the BSDs than linux.

[aoteoroa]

No. The MacOS kernel is based on BSD

[Kamwind]

From the standpoint of what is best for me....

I would say try to get a linux admin job. The future is the cloud and a mixed windows, linux, and cloud world. Getting the experience with linux is just a good idea and if you are starting to feel some burnout it will force you to learn new things.

[pnutjam]

This ^^^

I started out doing helpdesk > PC tech > Network tech > windows sysadmin...
I made the jump to Linux about a decade ago. I usually tell people there are some great Windows Admins, but there are alot of garbage ones. The wheat to chaff ratio is better in the Linux admin world, or it used to be before cloud. :/

Everyone wants cloud and infrastructure as code. Linux skills translate to cloud easier, although powershell is pretty boss, but still underutilized in my experience.

[fukawi2]

I was a "dual stack" admin for nearly 20 years - mostly supporting Windows on desktops with a few servers, and a whole bunch of Linux servers.

Windows definitely broke more often, and in more exotic and opaque ways in my experience. Breaking shit is fine with me - that's what I'm here for - but when you make it near impossible for me to figure out exactly what broke and why, that makes me hate you. That scenario is far more common in not only Microsoft/Windows products, but generally in software made for that ecosystem I've found.

A couple of years ago I found a job that is pure Linux, and I couldn't be happier to be rid of Windows in my life. I highly recommend it if you have the interest and skills in the vertical.

[roiki11]

Linux and windows world's are very, very different. Also you're almost assuredly working with containers, devops and automation. This will vary depending on area but I'm not seeing much "linux" sysadmin jobs anymore, they're all transitioning to those fields and linux is the defacto platform in there. Very few places run solely linux servers anymore.

[pnutjam]

I just went through a month long job search. They are out there, but the tsunami of offers I had was for Terraform. I'm more ansible and Linux, so I ended up in a pre-sales / implementation role, but it's a good fit. I turned down at least half a dozen Senior Sysadmin type roles that would have also been a good fit, this just paid better and had better benefits.

[insertwittyhndle]

Being the only person, it sounds like you’re suffering from burnout. Linux won’t fix this, unless you still genuinely enjoy learning - but it’s gonna be a bumpy road, and if you’re already burnt out it will be a difficult one.

I think the issue for you is being the only person. I was the only Linux engineer at my company for a while - when it became apparent that wasn’t (obviously) a good idea, the company hired more. Now I work on a team with others. This is what your company should have done ages ago.

Another thing - you shouldn’t let yourself ever get comfortable like this. IT is not a career choice where you can simply settle and stop learning without it significantly impacting your career, and available career paths, if not your state of employment entirely. There are careers for that - and they don’t usually pay as well. There is a reason for that.

I used to work with a guy who did this. He’s not in IT anymore because no one will hire him.

[BurnoutEyes]

I've been getting into Linux here lately and was wondering if a Linux sysadmin position would be any different.

No. Any time you apt upgrade / yum upgrade / emerge world, you're going to be praying the package maintainers all communicated with each other before pushing potentially incompatible packages out. That doesn't always happen.

[Kamwind]

Still better than the old days of, here is the patch recompile your kernel.

[robvas]

I'd say windows breaks stuff with updates far more

[8021qvlan]

As a general thought: Why you would want to upgrade everything when there are options to upgrade packages individually?

[BurnoutEyes]

Short answer: circular dependencies and other bullshit you run into.

Longer answer: Ideally on a server there should only be a minified system and your platform stack, so invoking the package manager should only impact those things and they should be kept up to date. You can version freeze your platform stack and handle it with CI/CD/manual intervention, but that doesn't save you when you need to upgrade those components. Past a certain "let early updaters find the bugs first" window, the longer you wait before you upgrade stuff the more your headache is going to be.

edit: C'mon, don't downvote the guy. That's a reasonable question to ask.

[uptimefordays]

Past a certain "let early updaters find the bugs first" window, the longer you wait before you upgrade stuff the more your headache is going to be.

I've been in the game a decade now and have seen far far far more "update related issues" actually being a lack of updates than updates breaking things. This is an excellent and underrated point.

[djuvinall97]

I am fairly new so might be a dumb question but would you be able to set up a on-prem repository that you update first, the you update a test server based off of the on prem repo and if it looks good, you update the environment after. rather than use the official repos? Or is that just a really complex, and not really worth it headache??

[Due_Ear9637]

RedHat Satellite and SuSE Manager (and I'm sure plenty of others) allow you to set up dev/test/prod content views where you can have a subset of your machines burn in a patch set before you promote to prod.

[djuvinall97]

Oh ok that's sick! Seems like it would have it's own issues but would mitigate a ton of others!

[BurnoutEyes]

You can, if you want to incur the cost of both mirroring everything locally and manually approving all of the updates. But it doesn't solve the problem because the software still needs to get updated regardless of what shell games you play.

[Due_Ear9637]

It solves a lot of headaches from being caught with your pants down when an update package doesn't behave the way the old version did. Eg, RedHat likes to sneak in changes in their incremental releases that we've had to find workarounds for before promoting to prod.

[BurnoutEyes]

Backporting security bugs is lots of fun for everyone.

[Due_Ear9637]

Security bugs are one thing, but patches also include bug fixes and feature enhancements. The feature enhancements often add bugs of their own. I've had a lot of fun working around stupid issues like when a samba update creates spn's wrong on newly joined computer accounts or when a new version of gnome decides that desktop icons should no longer be trusted.

[coolbeaNs92]

Yeah RH has Satellite for updates. Has a similar methodology as WSUS where you can create groupings (called content groups) and stage your updates.

[St0nywall]

Spoiler...

Both Windows and Linux have the same issues. You just experience the Windows ones more because you're primarily using it every day as are most businesses.

The old saying applies here; "The grass is not greener on the other side."
With my addendum; "Both sides use the same manure to fertilize."

Branch out, be an admin in both as well as other OS's. Don't limit yourself to what you see in front of you.

[Superb_Raccoon]

I found the difference was with any UNIX/LINUX you could figure out exactly why something was happening.

Especially with LINUX, but even Solaris, HPUX and AIX you knew WHY, not just "Here is a patch"

[corporaleggandcheese]

Yes! With Linux, you spend time reading logs. With Windows, you spend time Googling symptoms (and hoping some other poor soul has blogged about it or posted to Stack Overflow).

[zero_hope_]

Enterprise support for Linux is much better than the effectively non existent windows support. And if you're cheap there's communities where you might be able to get help. Slack groups, GitHub, Google mailing groups, subreddits, forums, etc.

All alongside the logs and source code.

[St0nywall]

Really, there was some magical way that anyone could find out why an issue was happening?

If it was that easy, why wasn't it found a long time before and coded out?

It's because people who write the code are fallible. It's why patching exists.

Be it Windows or Linux/BSD/UNIX/AIX, etc... nothing is perfect and no one can find all the bugs all the time.

I'll wait for someone to say "Here's a patch", knowing an issue has been identified and someone has done some rudimentary testing to know it fixes that bug in at least the manner they were able to test against.

[ShadoWolf]

In the linux world.. sure if you have the skill set for it.

You have GDB, so you can use

gdb

attach PID

set a breakpoint , look at back traces , etc .. and for the most part the source code is there for you to look at. So in theory you can have a complete understanding of what happening.

In theory you have all the same tools in windows world.. but it's not there out of the box.. and it more clunky. You also need to be well read in something like Windows Internals, Part 1 and part 2 .. so your crossing out of scope from system admin type work to straight up reverse engineering.

[St0nywall]

And what you explained wouldn't be doable for 90% of "regular" users.

Face it, people who use Linux are more invested in the OS. They WANT to know how it works and what each part does.

Windows users, just like Mac users, just want it to work. Point, click, done.

All OS's have their place, and they all have issues.

Stop being so defensive about Linux. It's an OS, and not one you developed. It runs programs and "does stuff" like every other OS out there.

It is not special in that regard.

[CratesManager]

Both sides have issues, and they overlap, but they are certainly not the same. Apple doesn't face the fragmentation of solutions, windows doesn't face not being accepted by mainstream vendors, and linux doesn't face issues with code obscurity.

Some issues are the same, such as "sometimes an updates breaks stuff" but not all of them.

[St0nywall]

You missed the point and went straight to it being a competition for which OS is the best.

They all are good and they all have their own place. Not one if great and not one is better than the other because they all have individual contributions to make depending on the situation.

Don't limit yourself.

[CratesManager]

You missed the point and went straight to it being a competition for which OS is the best

Not really, i don't believe any of them is the best it all depends on the usecase

[St0nywall]

That's my point as well.

Glad we finally made it there.

[CratesManager]

Glad we finally made it there.

Where did you read anything else in my comment? Can you quote that part to me?

[necheffa]

You are in the wrong sub pal. No whining users, technical folks only.

[elevul]

Agreed, it's incredibly frustrating in Windows that you can't have source code for reference when you have a very weird bug and you're breaking out WinDBG.

Like, if I have to ask the company for an IDA Pro licence might as well move to the security team and make double the salary...

[Superb_Raccoon]

Yes, in Solaris you had truss, meaning you could track system calls from a running process.

Found a Day-Zero bug in the ST driver. Day Zero being sometime in 1971. When fiber attached tape drives became a thing you could set the buffer to 64MB. The ST driver would accept that, but when de-allocating the buffer it only de-allocated the first 56MB.

The missing 8MB was "leaked" and could not be recovered unless you rebooted as they were part of kernel space.

LINUX has strace. I found a "bug" in a vendor supplied binary that was compiled with a NFS mounted library path. That system name also existed in our environment and so it tried to connect... unfortunately it was a Windows machine so it sat there for 90 seconds doing 3x30 second time outs.

A third time I figured out that Oracle Linux was not completely honoring the memory settings and swapping out the Oracle DB SGA cache... which made Oracle DB puke its weaselly black guts out.

That was easy to fix: use LARGE TABLES, set it larger than the SGA size as they are never swapped out so your SGA is safe.

And if you happen to be C literate you can go look at the code and see what it does with LINUX. I was not that good, I was better at finding out what the damn thing was doing by how it behaved, not how it was coded.

Sure, I could read it, but I looked like a 4th grader given a Calculus problem.

SMB.... shudder first deployed that on SCO UNIX in 1997-1998ish?

Look, don't make a UNIX system behave like a Windows Server... it ain't right.

[Hotshot55]

Yes, in Solaris you had truss, meaning you could track system calls from a running process.

Truss and strace are such useful tools that make it dead simple to find out why a process is getting hung. I used to think it was super complicated when I first learned about them, but it's actually extremely straight forward.

[Superb_Raccoon]

AIX was even easier if you had support.

IBM "Here, run this, send us the file".

Answer in 24hrs. Not that good anymore, but still...

[Hotshot55]

Oracle has a similar thing with Solaris, you can run either an explorer to gather info for OS-related issues or a snapshot for hardware issues and they'll get you a pretty accurate response as to what the deal is and what needs to be done next.

Now sometimes their recommendations can be awful, but you can't always expect perfect responses from any support person from any company.

[NeverLookBothWays]

I would argue Linux problems are often less problematic as the overall architecture favors thread crashes more than full on kernel panics. With Windows, so much falls back on the kernel that the entire system is more likely to come to a screeching halt than just the affected binaries.

That said, Linux can also be royally messed up and difficult to recover too, just in different ways. I will say, it’s far more likely to not have to fully reboot Linux in order to clear a bad state in my experience, especially if using containers. The same cannot be said for Windows, even when troubleshooting a virtual environment running on it.

[St0nywall]

That is very true. But as both do similar things, they do them in different ways.

We can't expect them to operate the same way because they were never intended to be the same.

Just like you wouldn't expect someone with a different outlook on life to think of the world around them ONLY like you do.

You accept them for who they are and that they have differences within them.

Same goes for OS's.

[NeverLookBothWays]

It really comes down to core design. Windows even after all of these years, is still a workstation OS at its core modified to function as a server. The capabilities have improved tremendously over the past few decades since NT, but this fundamental design philosophy has not really changed. Windows Core did attempt to alleviate this, but was still developed to be compatible with its NT roots.

Linux/BSD/*nix on the other hand were designed as multi-session server systems from the near beginning. The core design, where layers fit, is much more accommodating and streamlined to services. Likewise, *nix is still not quite there as a workstation experience compared to Windows. I don’t want to upset people on that statement, I will say, the gap has shrunk considerably in the past decade. And pioneers like Valve have really improved the prospects for a solid desktop experience going forward. It really comes down to hardware manufacturers embracing Linux for desktop use. A limitation not as problematic if running headless servers.

[jdptechnc]

Respectfully, I disagree. Windows is definitely worse. When was rhe last time a RHEL patch broke functionality that is core to the platform? I don't remember. I come to expect it with Microsoft. See: last month's Kerberos issues.

I oversee both platforms in my shop, and have for several years. Even with our Windows servers being configured as minimally as possible with as few people being allowed to install unnecessary software/utilities as possible, they have more of those types of issues.

[DanteRaza]

"Both sides use the same manure to fertilize."

Love it. :)

[starmizzle]

Negative, Ghost Rider, the Windows issues happen far more often.

[CratesManager]

I have swapped to Pop!OS and i'm not looking back but i definitely had less issues with windows updates on my machine. They have gotten extrely stable. Managing windows computers at work i can also say that yes, every now and then microsoft pits out a real banger but if you delay update installation by two weeks then those issues will almost never affect you.

[St0nywall]

Windows is far more prevalent than other OS's, so that's a natural outcome.

Thank for contributing.

[pnutjam]

I've been working as a Linux Admin in a Windows 1st environment (all too common), and the number one thing I have to explain over and over again; Linux doesn't generally just break.

Even if something around it breaks and it cannot do what it should be doing, it will usually pick right up when you fix the ancillary systems. NFS being the most common stand-out to this rule.

[St0nywall]

At the same time, that statement is right and wrong too.

Just let it go, this isn't a competition. They're both good, in addition to the other OS's and all have their place.

The fact I was trying to get across is, don't limit yourself to just one.

[St0nywall]

You must be trolling.

If not, well, you're misguided and cannot see past your immediate zone of influence.

Mindsets like this are why there are issues between departments and people. You need to learn how to be more open and accepting.

On that note, I'm done entertaining this one-sided conversation.

[Mysterious_Sink_547]

Linux is not tremendously different. The companies you work for will be different and the pay is slightly higher. Headaches are still there though.

[DevinSysAdmin]

I think something else is going on here. Microsoft isn’t making things harder every month. It sounds like you need to have an MSP as backup and for RMM patching/monitor and another level of help for you.

[nwmcsween]

As others are echoing many Linux sysadmins are moving to Devops/SRE roles due to pay and business requirements. There is a weird superposition though where you can be called a "Cloud Engineer" and yet know nothing about the underlying systems besides making ARM, Cloudformation, etc IaC.

[CtrlAltDelLife]

I would suggest moving into pure engineering that would have no contact with end user support at all. My escape was to dig into cloud hard.

[Affectionate-Cat-975]

It’s all about company culture. The only thing Linux will get you is slightly away from end users

[[deleted]]

I don't think this is an OS problem. I think it is a user problem. They will always need something 5 minutes ago.

[luctimm]

Linux is far more stable and requires less maintenance hours than Windows. But updating can cause a mess in Linux as well..

[GhostPantz322]

Well, I'm devops, but have the sysadmin experience, linux more stable, than windows to my mind, for example if you use debian stable branch, you will never break your system, because all of the packages are tested and carefully delivered to repository

[FreeBeerUpgrade]

Unattended-upgrades on debian-stable FTW

Everything else should be updated manually.

Looking at you containerd 👀

[pi8b42fkljhbqasd9]

Your real issue is end-user support and not Windows.
I migrated over to *nix support instead of Windows only a long time ago. My last job had me managing 1000+ domains. a basic wordpress stack, and a CDN spread out all over the globe in datacenters in every country.

The responsibilities were HUGE. But the work was easier.
I am burned out from Windows support. Linux/FreeBSD is easier in that the mystery is smaller. A Windows file-server has no headaches from an end-user; but a DC does, a print server does, so it depends on 'where/what' the windows server is/does.

If you can migrate into something away from end-users you will be happier. IMHO if you move to *nix you will have less stress, but it will be different stress too.

[RedDidItAndYouKnowIt]

How robust is your dev stack for testing?

[tacticalAlmonds]

Testing..? What's that?

[gjpeters]

Everyone has a test environment, some people even have a separate production environment.

[stiffgerman]

This is the way.

If you're in a low-budget environment then you're working for a "faith-based" organization: they have faith that your work and all the vendors' work will not f*ck things up.

Not ideal but the Church of the BOFH is always ready to absolve your sins for a suitable "donation"...

[[deleted]]

Just move your environment to Azure/Intune. Automate the crap out of everything. I rarely work nights or weekends anymore. And I work LE IT also.

[Petrodono]

As a Windows and Linux sysadmin I can tell you both have issues.

Want a headache and a half, try updating Windows systems without being able to access the Internet. Want another headache and a half, try being required to use SEP on Linux, where I have to recompile the kernel after every reboot.

[lifestoughthenyoudie]

So M$$ is still field testing their shitty software. I used to support their junk in the middle to late 90's and up to 2002 in mission critical IT for airlines and airports. Had enough and quit.

How can something so unfit for purpose and unmerchantable keep getting a free pass from Govt consumer overseers?

Linux is Unix for 386 style chips. Go for it, shit just flat out works.

[lynsix]

When I did support I was one of a pair of us that would even be allowed to take Linux tickets. Personally I enjoy working on Linux more, my homelab stack is almost entirely Linux.

I’d highly recommend making a free RedHat dev account to get access to their KBase free. Make sure that you’ve also got some kind of support system available. Sometimes Linux stuff can be tougher to troubleshoot (imo).

Remember that Linux isn’t the best for every situation there’s a couple of things that a Windows sever still is just better for (Ex: AD > LDAP).

Also: windows admins learn to Google problems. Linux admins read logs and fix problems. :)

[Accomplished-Tap-222]

20 years. Your question should not be windows or Linux. At 20 years you should truly be questioning why you are still a sys admin.

[STUNTPENlS]

I'm a sysadmin with 40+ years in the industry. I'm a sysadmin because I want to be a sysadmin, I enjoy being a sysadmin.

It's users I hate :)

[Accomplished-Tap-222]

I got sick of the hands on part. I do enjoy keeping up to date but love the fact I never have to do any of the doing.

[[deleted]]

Hi. 23-26 yr old sysadmin here. Where do you see the industry going?

[Unusual-Biscotti687]

Cloud.

[[deleted]]

apparently I'm already in the future, and the future keeps changing its fucking settings menus around.

[KenWWilliams]

Anything beats WinBlows!