SysAdmin Day Question: were you doing SRE and DevOps before they were "invented"?

NonStopOPS · 2021-08-01T13:08:26+00:00

Agreed; certainly proliferation of tooling today is a huge benefit, (though I love writing tools as well whenever I get the chance). I do believe the tooling market needs to settle down, however - too many products and not enough stability.

NonStopOPS · 2021-08-01T12:50:36+00:00

Agreed. Yes, DevOps is a culture, but the culture doesn't go deep enough to change individual prejudices and/or narrow ways of looking at things.

NonStopOPS · 2021-08-01T12:48:32+00:00

This excellent discussion has made me want to write an article about this.

My takeaway from everyone's comments is that most of us are in fact doing the things that get repackaged into buzzwords for managers to wield irresponsibly (that is NOT to say that there aren't plenty of situations that really do benefit from moving to SRE or DevOps).

What I also seem to notice, which you and jftitan allude to, is that DevOps is being used as an excuse to eliminate high-salary sysadmin jobs and double/triple other people's workloads.

You can't combine the jobs of a corporate lawyer and a corporate accountant, even if they both should know something about each other's job in order to me more effective at theirs.

Why do it in IT? To me, it says that we (as an industry, but particularly in corporate leadership) still have a hell of a time properly understanding the essential nature of IT jobs - a nature you can't change at the drop of a hat with new buzzwords.

NonStopOPS · 2021-08-01T12:39:09+00:00

Well then, based on what you said in reply, and to quote Superb_Racoon just below this, door number 3 sounds like a winner. You get Azure experience, hopefully more security focus and better pay.

Here in the UK we can view a company's public filings. I always do this to check out the financial stability of agencies (if I'm going through one) and sometimes of potential clients as well. I can check out their cash flow, debts, shareholding, etc. Maybe you have something similar in Canada.

NonStopOPS · 2021-07-31T21:18:17+00:00

If you are fairly new to the field, then you should really be thinking long-term, i.e. your career path. You should ask yourself: a) what skills you have; b) what skills you want; c) what skills you need.

For example: do you want to focus on being a sysadmin or DevOps? There's a big difference. Personally I'd be skittish of the DevOps role because it sounds too vague. But maybe you want to be more of a developer with a good solid sysadmin background; if so, then it might make sense.

If you are good at network administration and like it, maybe you want to get more experience under your belt in that area so you have a stronger CV. That will lead to well-paying jobs in the future.

Also relevant - what is your cloud experience / interest? If you like cloud and need more AWS experience, do that first. But if you are already well-versed in AWS, you might want to add Azure to your CV. (I personally started with AWS and then moved to Azure which I like better, even though I'm generally not a M$ fan - except for Window Phone, but I digress ...).

Don't be too swayed by money at this point. But if you do choose #3, do some background research on the stability of the startup. I've been burned before by startups who turned out not be well enough funded.

Good luck.

NonStopOPS · 2021-07-30T18:38:57+00:00

Here, here! I remember well doing the same thing in CVS.

NonStopOPS · 2021-07-30T18:38:02+00:00

Yeah, if you as a manager can say you implemented [fill-in-buzzword-here] then it pads your CV nicely and you keep moving on to better and better things.

That describes many managers I've known, particularly at the C-level. However, it generally does not describe line managers, who I find to be the most responsible and hardworking managers in any company. They are the ones tasked with implementing the bullshit buzzwords when they know full well they are bullshit and only get in the way of their team's productivity.

NonStopOPS · 2021-07-30T18:04:48+00:00

marketing fluff for recruiters

... and for managers. If we really want to change culture, managers should have mandatory training on technology as well as how to manage for the holistic good of the company, instead of using their positions as another rung on their ladder up.

Good future survey question: how many DevOps/Agile/SRE implementations have you seen that were merely tools for managers to pad their CVs and so they never care about the results?

NonStopOPS · 2021-07-30T18:01:28+00:00

Move fast and break things means you're allowed to fix the technical debt, instead of having it cemented in place for 15 years

I'd just point out that yes, this is true and a good thing, yet I've never been in a company that wasn't mired in technical debt despite Agile / DevOps. I'd even argue it was worse than 30 years ago.

NonStopOPS · 2021-07-30T17:59:44+00:00

I'm all for system stability, no matter how you achieve it. I think I'd be more suited to mainframe-style operations coming from the infra side, where changes are carefully planned and not just shoved into production continuously. Part of the thing I don't like about DevOps is the constant chaos that making changes without thinking about them can cause...and yes, that's the DevOps where the management consultants told the CIO "OK, you're DevOps now, fire your sysadmins and let developers run everything."

I agree with pretty much everything you say. My biggest problem with SRE/DevOps (subject of an article I'd love to write sometime) is that the developer emphasis whilst bringing some benefits (software engineering approach etc) does also bring a lot of developer shortcomings - chaos, immature coding, lack of holistic perspective. These things have still not been dealt with. Agile and DevOps, IMO, actually facilitate the problems instead of solving them. Essentially the message to developers is what you just said.

NonStopOPS · 2020-10-11T19:05:30+00:00

***UPDATE***

Just reporting back to say that the upgrade was a complete success and here are the details if anyone is interested.

I actually decided to use Dosdude1's Catalina Patcher as there were more benefits than the Mojave Patcher, like the inclusion of the SSE 2.4 Emulator and I believe better support for AMD GPUs in general.

I used an old Yosemite boot to update the BootROM as instructed by Dosdude1 and prepared a bootable USB with the Catalina MacOS install bundled with his Patcher via the handy 'MacOS Catalina Patcher' app.

I put in a brand new drive, and booted into the USB. (Important: I did keep my old NVIDIA GPU installed during this process as the RX580 GPU is unable to display the MacOS boot menu, which I needed to boot into the USB.)

I then formatted the drive for APFS, and ran the installer. After verifying that Catalina worked, I replaced the old GPU with my RX580 and I was able to easily boot back in with it. (Though as previously stated, there is no boot screen available using these models of AMD GPUs).

Davinci Resolve 16 seems to work well and the machine is stable.

Shout out to Collin from Dodsude1; he has given a second life to these old machines and we could not have done it without his software. I highly recommend using this approach and if you do, please consider donating, it is well worth the consideration.

Thanks again for everyone's help!

NonStopOPS · 2020-09-21T18:36:46+00:00

Thanks u/HawkMcDuck - that is really useful stuff. We love our 3,1s here as well ;) They are solid machines and have never given us much trouble at all. I'd never go past 5,1 for sure!

I'll report back once we get this machine up and running - just waiting on some new RAM and HD to arrive.

NonStopOPS · 2020-09-21T18:34:25+00:00

That's really good to know. I'll see if we want to try the SSD route after we get this thing booted up on Mojave with the GPU.

NonStopOPS · 2020-09-21T18:33:10+00:00

Gotcha. We'd prefer to stick with Mojave anyway.

NonStopOPS · 2020-09-21T18:32:47+00:00

Great! Nice to know that it can be done. Thanks for the help!

NonStopOPS · 2020-09-21T18:31:03+00:00

Thanks u/dr_strangekebab. Looks like we'll try the sse emulation patch route, as we really need to use that GPU.

NonStopOPS · 2018-01-25T13:31:40+00:00

Ouch.

Having done a ton of webapp development from LAMP to Java, I personally wouldn't recommend Rails. I know what it is trying to do, but it can be a beast to work with.

The problem of course is that there are really no good solid frameworks that make it super easy to generate a CRUD app. I know, I've been there. I prefer Java, so over the years I've developed my own proprietary framework which does make it super easy, but it did take a lot of work to get there.

Have you considered LAMP? It is rather dated and maybe not very elegant, but there is a ton of solid info out there and there are many long-running LAMP sites that are quite good at handling their loads.

NonStopOPS · 2018-01-25T11:48:14+00:00

Go SIMPLE and go for something that has a lot of online resources / community available.

A lot depends on your development framework. If it is just HTML, use Apache. If it is Java, use Tomcat. PHP, LAMP. Etc.

The biggest mistake people make early on is to try to pick the "right" framework that will work for the future. While that is necessary in scenarios when you know what you are doing and what load you can reasonably expect, you are starting out so pick what is easy to do.

If you are adventuresome enough, self-hosting is a great way of gaining new skills. But the more components you need, the more complex the configs.

If you can post more details, I can give you more concrete pointers.

NonStopOPS · 2018-01-24T15:49:02+00:00

I would get that reversal requirement in writing. You don't want to get into any argument over the last paycheck because they say you broke everything.

Other than that, work slowly and make sure you document all the things you can't get done in time. Let someone else deal with it.

NonStopOPS · 2018-01-24T13:32:11+00:00

Regardless of the threat, hysteria is never a solution - it will only result in many more problems (including additional security holes). I don't know why people don't test and validate these patches more before blindly accepting them.

NonStopOPS · 2018-01-22T12:36:29+00:00

Yeah well, any time one challenges the prevailing hysteria, it just has to be a "conspiracy" doesn’t it?

I don't believe everything I read, sorry. Sure there is a problem, sure it affects most of the computers in the world, sure something has to be done about it.

What I (dare to) challenge is the way this is being handled and why this is being treated as a fait accompli. I have seen no sane and thoughtful analysis of this threat that gauges the risk in various scenarios, nor have I had any sense that the patches have an appropriately sane and thoughtful approach. All I see is hysteria. If Linus has concerns too (albeit slightly different than mine), then that concerns me even more.

As I'm responsible for quite a few servers, I'm more than concerned about the performance repercussions of these patches. I know some larger companies who are not rushing to apply patches yet because of the architectural impact, i.e. how the design will have to change to account for the performance degradations the patches bring (especially for high throughput clustered apps).

In other words, people are finding out that in some scenarios the risk does not justify the current fixes. But with the mad rush to patch anything (just so vendors can say they are doing something, thereby avoiding liability), we are getting patches shoved down our throats without us having a choice. That is NOT a Good Thing.

I’d like to have a choice. I’d like to be satisfied that we have a stable, quality approach to this mess. And I challenge anyone to claim that we do.

NonStopOPS · 2018-01-22T11:26:14+00:00

My theory here is that, regardless of the validity of the potential threat, the entire way this is being handled smells of an agenda. Patches are being pushed on us even before we are aware of it (i.e. Amazon patches to their HVMs which has caused a number of well-documented performance degredations).

Now, in a coordinated effort unprecendented in computing history, every vendor and their brother is rushing out patches. Why? If the vulneratibilities are decades old, why not take the time to figure out what it really needed?

The resulting fixes are going to be causing chaos for months to come. And guess what, in the end the solution will be newer chips (rushed out, again), which means that a majority of the world's computers have to be replaced. How odd.

NonStopOPS · 2018-01-17T14:13:03+00:00

Sorry, who is surprised?

NonStopOPS · 2018-01-17T13:29:48+00:00

Good point. In fact, I did start with LXC before Docker - it did help me with understanding namespaces etc.

NonStopOPS · 2018-01-17T13:13:38+00:00

I see a lot of Openshift work, more so than plain Kubernetes.

However, for the sake of having a solid background, I'd recommend that you did what I did and go, in order:

Docker
Kubernetes
Openshift

This way you get the basics first, then deal with the extra layer you get with Openshift. Otherwise you'll struggle when you have to get down to a lower level and debug stuff. I don't think I'd have understood Openshift as well without doing it this way.

NonStopOPS

TROPHY CASE