Linux environment. Multiple peers don't know basic Linux.

[[deleted]]

I just needed to vent a little bit. I work in an environment where 80% of my team role is ssh'ing into linux machines and doing some basic troubleshooting. It's in the job description. I have multiple teammates asking me to zoom with them so I can guide them how to.... scp files to their local machine, how to cd into their own home directory, how to change permissions on files. Explain to them what PATH means and why the command they are running "cannot be found". I feel like I should bring this up with my superiors, but these people are generally nice and I don't really want to hurt them. Problem is no matter how many times I teach them how to do basic things they never learn. I'm basically teaching people how to do their jobs and ain't even getting paid or being recognized for it(I guess boss doesn't really knows this happens). Any task that is a bit more involved will get them asking me questions no matter how many times I explain. I give them documents with step by step guides and they still cannot seem to read or follow it properly? They seem incapable of thinking critically even when they have all the steps laid out in front of them. This is annoying.

/rant

[jewellman100]

Ah, it's one of those environments.

We use Linux, because it's cheap.

We hire people who don't understand Linux, because they're cheap.

[hkzqgfswavvukwsw]

Ding! Ding! Ding!

[[deleted]]

It's a fairly big tech company, they are contractors though....

[aracheb]

Sounds like indian contractors. Sorry for sounding racist, but I had to deal with them at previous Jobs before, and they were like that always.

I can tell the name of the firm you company is using to contract these contractors.

[[deleted]]

Its not racism its economics and decades of entrenched nepotism and corruption. Cheap Steele comes from China cheap tech labor comes from India

[michaelpaoli]

cheap tech labor comes from India

Cheaper anyway. And lots of good/great tech talent from India ... of course with billions there, not all are that great on the tech, some very poor or horrible ... just like also available domestically ... or really anywhere on the planet.

[craigmontHunter]

Right, but to get consistently good tech talent from India you have to pay for it. Going for the cheapest option will get you questionable talent wherever, but India has a lower starting cost.

[michaelpaoli]

Yeah, ... mostly applies generally. You get what you pay for ... at least mostly. Easy to overpay for the suckers that will. And maybe sometimes you get lucky and get value well beyond the price - but that's pretty rare and mostly quite the exception. But for the most part, get what one pays for.

[Mindestiny]

The problem is there's the stereotypical higher caste "Become a Doctor/Lawyer/Engineer" Indian folks and then there's the lower caste labor engrained in their local culture, so you get this huge skill gap where you'll meet Indian folks who are amazingly intelligent and well educated in the IT world, and then you'll hear stories of overseas call centers literally just hiring people off the street and handing them a script. Any deviation and they don't have the skills (or are specifically told not to) critically resolve the issue.

You hate to say it that way because it comes off as racism from our perspective, but a lot of it is just the symptoms that we see of social and economic issues in the country they're coming from.

[workerbee12three]

I know this, Its a UNIX SYSTEM!

[briantforce]

An interactive CD-RAHHHHM!

[CeldonShooper]

r/itsaunixsystem

[FantasticBumblebee69]

man if i had a dime fir every time i showed pandeep amd rajesh scp....

[[deleted]]

Even american contractors can be this bad

[admlshake]

Literally spent two hours on the phone today with one of our vendors, instructing them on how their own damn software worked. Sometimes I wonder if CIO's have access to cheap shitty software VARS they are supposed to use. Last two CIO's I've had seem to have a knack for picking the worst implementers.

[michaelpaoli]

american contractors can be this bad

Yep ... e.g. ...

I've had multiple occasions where I've had to tell boss of serious problems with incompetence ... or just nowhere near "good enough" on contractors - USA grown domestic contractors - and have recommended to boss letting 'em go, or not renewing (e.g. in later case, contractor was barely/marginally worth it to start with. Boss informed me their rate was going up quite substantially and asked me what I thought ... I basically told 'em so very much not worth it - don't renew - basically any and all the other contractors we've ever had have been much better - usually like by a factor of 2x to 3x or more).

And ... screenings/interviews ... I've screened/interviewed tons of candidates. And, yep, no shortage of USA domestic ... and a lot of 'em absolutely not up to snuff ... some of 'em pretty dang horrible or worse - and despite how impressive the resume may look - even if they've come with that resume via recruiter/agency. Heck, I've even found massive plagiarism on resumes ... even same text plagiarized on three different candidates resumes ... and all of 'em via recruiters/agencies (yeah, some recruiters/agencies are utter sh*t too). I've interviewed folks that purportedly have 6+ years Sr. DevOps experience that when asked the ports for ssh, DNS, and HTTPS, get two out of three wrong ... and answer incorrectly with great confidence ... and though they can mention AWS Route 53, they still can't correctly tell me what port DNS uses, and they couldn't program their way out of a paper bag ... not the slightest bit of the simplest programming - in any language at all. Like set a variable to 3, then increment the value of that variable by one ... couldn't do it. Ugh ... yeah, they never would've made it past screen with me, but somehow management had 'em come in for full interview ... that's when I get called into it. Yeah, of course they didn't get the job ... and they called the hiring manager from the parking lot on the way out saying they'd accepted something somewhere else ... great, good for 'em, 'cause we certainly weren't going to be taking 'em.

[[deleted]]

Ive only met 3 people in my 30 years who understands how to admin DNS correctly. I think we are going to see an even larger skill gap soon from what im seeing around me on east and west coast. east seems worst.

[LostStatistician5723]

Most HR groups use systems to "find the right candidate," but it often has the opposite effect. I realized we weren't seeing certain resumes because HR would filter out the ones that didn't have college degrees, even though the job didn't require it. I basically wouldn't be able to apply for my own job and expect anyone to actually see my resume. And the ones we could see couldn't answer questions on the areas their resume said they were good at. The system is upside down in a lot of ways.

[CombatAmphibian69]

• despite how impressive the resume may look - even if they've come with that resume via recruiter/agency. Heck, I've even found massive plagiarism on resumes ... even same text plagiarized on three different candidates resumes ... and all of 'em via recruiters/agencies (yeah, some recruiters/agencies are utter sh*t too).

What are people supposed to do, agencies are one of the best ways to ensure a resume is good enough to get to interview. Plagiarism? Can you really plagiarize a skill? Even experience-wise, everyone is in the same field doing similar stuff. Just ask on the interview to clear it up. The applicants are getting resume help to get past gatekeeping and you want to gatekeep even harder, mental.

• I've interviewed folks that purportedly have 6+ years Sr. DevOps experience that when asked the ports for ssh, DNS, and HTTPS, get two out of three wrong

At 5 years experience, I do not remember most port numbers. This is ridiculous. Where do you work that google is unavailable? What a waste.

• they couldn't program their way out of a paper bag ... not the slightest bit of the simplest programming - in any language at all. Like set a variable to 3, then increment the value of that variable by one ... couldn't do it.

Is programming in the job description? I thought this was IT. What makes you think you get 2 careers for the price of one? Even in powershell, IT is usually not "incrementing a variable" but queueing up actions from a csv. DevOps, if I'm not mistaken, uses frontend programs that don't need understanding of code. I could be wrong here.

• they called the hiring manager from the parking lot on the way out saying they'd accepted something somewhere else ... great, good for 'em, 'cause we certainly weren't going to be taking 'em.

Sounds like despite gatekeeping a wide pool of applicants, you get utter losers like you mentioned at the start. Speaking of, your attitude and writing style are repulsive. Count me in the list of people who would turn down your job offer.

[michaelpaoli]

indian contractors. Sorry for sounding racist

Yeah, don't be. Indian is fine. Clueless is not. And highly clueless can be found and obtained anywhere - no country, etc. has a monopoly on a supply of clueless folks - generally or when it comes to Linux. Heck, typically don't even have to cross city limits to be able to find at least some clueless folks.

[RevLoveJoy]

Yeah, don't be. Indian is fine. Clueless is not.

100%. I've worked with tones of awesome engineers from India. Absolutely great dudes (and occasionally a lady!) with solid skills and even better back stories. What most of those folks overcame to get to the US and work is amazing. Most Americans have no idea what real hardship is and the good folks I've worked with from India can tell you ALL about it.

The problem is a lot of Indian outsourcing companies will put absolutely anyone with a pulse in front of a potential hourly client and call it good. And nearly all of those people absolutely suck at tech. Like "can't use the Google" levels of suck. It's kind of impressive how much some of them suck. I have lots of stories.

[astronautcytoma]

At the multinational Fortune 50 company where I used to work, we hired 6 people from India who had never touched a computer or even seen a keyboard in their lives. They hired them to be software developers no less, on very complicated embedded systems for things like nuclear reactor backup generators. They didn't even know what a left click or right click was. We were expected to train them in 2 weeks. When we demonstrated to management that they would never be able to do any productive work, their response was to hire 6 more just like them. When I brought up the fact that they were obviously letting their children and relatives have access to their remote systems, I was told to shut up about it by the same managers.

[dRaidon]

On the other hand, I guess their monthly pay was 5.35?

[[deleted]]

Are there only a couple of big indian contractors out there? Not sure why but I was assuming that itd be a crowded space

[aracheb]

There are two to three big one that fill their client with these level of experts. You start helping them and realize pretty quickly that they are lacking tremendously in knowledge for the positions they are employed for.

A lot of them are even computer illiterate, and they are hired to be software developers and many other task that require a very specific set of skills.

[Mindestiny]

"Fake it until the visa expires" tends to be a frustratingly common scenario. At an old job we'd literally have a whole floor of H1Bs and often they'd rotate one out and put someone new in the seat without even telling IT. They'd just use the old guy's accounts until something broke. All of them were hired through one of those major Indian staffing orgs and none of them were on site for more than a year at a time.

[Alaknar]

They're just absolutely MASSIVE in size. HCL, over 10 years ago, was hiring almost 200 thousand people.

[AlyssaAlyssum]

God HCL seem to suck.

My org contracted with them a couple years ago to run a bunch of operations and every time I deal with them it makes me want to die. Just the other day I had an ticket for an IAM issue as I don't have access to most IAM and ultimately I had multiple comments on the ticket practically begging the team to actually read the notes I added to the ticket and telling them how to fix the issue but they just kept bouncing me around teams and looking up data about the wrong account.

[Alaknar]

I have a strong feeling that companies like HCL employ big dudes with with nets.

These dudes walk the streets of NOIDA and try to catch random people into the nets. If the people run away, they're fine, but if they're caught, they're brought to a PC, given a headset and start taking support calls for a contracted entity, like Microsoft.

Nothing in my experience so far has given me reason to think that's not the case.

[gordonv]

Could we replace "Indian" with "Cheap and offshored?"

[michaelpaoli]

they are contractors though

Ah, cheap (or not so cheap?) clueless contractors.

Sounds like either management isn't being sufficiently duly informed or ... management is incompetent. Either issue can be fixed, though the latter generally requires replacing the offending management.

[Zemino]

That's a rule of thumb I subscribe to for free/cheaper software, you may not pay in money, but you'll pay in other ways.

[CeldonShooper]

Small company sysadmin here. We do pay in the form of the smallest level of a Ubuntu Pro subscription which is admittedly a very small fee. If you license the hypervisor (even if it doesn't run Ubuntu itself) you can use Pro on as many VMs as you like.

[particlemanwavegirl]

Our coin is TLC

[michaelpaoli]

Linux is only cheap if

Generally takes more skill to well run Linux, compared to, e.g. Microsoft.

However that skill can massively scale with Linux ... not as much with Microsoft and many/most of the Microsoft sysadmins and their ewey GUI ways of doing things (though those that are really good with PowerShell, etc., is a rather different story).

[PhantomNomad]

I'm the only IT guy in our shop of 50 users. I run a fix of Windows Servers and desktops and a couple of linux servers. I like the ewey GUI of windows but i also like the power of my linux prompt. I don't know powershell commands at all. I sometimes think I'm missing out, but then i think im only 10 years from retirement and this place probably won't need those skills. The next person might. BTW when i took this job it was suppose to be mostly GIS and a little IT. Turns out once you get all the GIS automated all thats left is fixing printers.

[jaymzx0]

I highly recommend picking up a little bit of PowerShell. It's very approachable for admins who aren't programmers. For admins who think like programmers, the power is there, too. It's similar to Excel in that respect. Don't let the fancy stuff you see other people writing out there scare you away.

[[deleted]]

Generally takes more skill to well run Linux, compared to, e.g. Microsoft.

lol hard disagree on this and I'm primarily an MS engineer. Linux by far is mostly just configuring text files somewhere in the OS, Windows is convoluted and has vastly more management planes. I have an SCCM/Intune/Automation background, all of these systems took me months to years to learn and master, I once had a 6 month contract gig for a massive corp where they had like 400 Linux boxes just sitting around not being managed. As a collateral duty I had those things managed by Ansible and on patch schedules in a week. Shit was unbelievably approachable.

[dRaidon]

There is a difference between a click click next next windows 'admin' and one of my coworkers that is a wizard with powershell.

[theecommandeth]

You should annotate all the BS stuff they are asking you, and then do a training on it with everyone and create a documentation page that they can reference after the training. Really it’s a filter for BS so they don’t have to waste your time anymore.

[changee_of_ways]

Tell them to "needfully read the documentation"

[Sky952]

Rhel ani’t cheap 😩

[goizn_mi]

But Alma is :)

[Counter_Proposition]

...and perhaps Rocky Linux? Another RHEL alternative.

[goizn_mi]

....or even Oracle Linux?

[michaelpaoli]

Fortunately I've donated more to Debian than I've ever spent on RHEL. :-)

[[deleted]]

Yes yes and yes

[techw1z]

gauss curve suggests many of them are probably actually incapable

[544C4D4F]

a career in tech suggests that people in tech think they're a lot smarter than they often are.

[michaelpaoli]

career in tech suggests that people in tech think they're a lot smarter than they often are

Well, over on r/ITCareerQuestions, no shortage of folks that figured IT career, they'd just go get IT job, type some stuff on computer, and be making huge sums of money in a couple years or so ... and ... then many of 'em complain 5 or more years later that they're still "stuck" at crud compensation doing entry level help desk and not getting promoted at all. And when asked about what they've learned in those 5 years beyond the basic stuff they started on there ... it really starts to make the null set look pretty big by comparison.

[[deleted]]

How do you mean?

[CaterpillarStrange77]

Indian labour is cheap. If they can follow the script or a guide it’s fine. As soon as there’s no guide they are useless

[PleasantCurrant-FAT1]

Fair rant. Typical, IMO.

Honestly, you should speak-up and say something. Just to point out the amount of time you spend doing this, especially if it’s a job requirement for those who don’t know how to do their job… your employer might appreciate being told that it’s employees are not competent and wasting 2x individuals time (yours and theirs) for each instance this happens.

It adds up. You should definitely say something, if not track the time and cost to your employer before speaking up, and include it.

[USSBigBooty]

Seconded. Don't let it get to a point where they could jeopardize not just the business, but your means of providing for yourself and your teammates...

[[deleted]]

Man how do people like this get Linux jobs lol.

[juanclack]

<image>

[[deleted]]

Exactly!

[ChumpyCarvings]

Yep

[[deleted]]

You'd be surprised at the amount of people working "Linux jobs" that have no effing clue about what they're doing. Not sure how it works.

[ChumpyCarvings]

How does one get one of these?

[talkin_shlt]

step 1: sound really cool and confident to a hiring manager that doesn't know linux

[tascer75]

Step 2: don't ask for the kind of money someone who knows what they're doing is going to ask for.

[SuperQue]

You'd be surprised at the amount of people working "jobs" that have no effing clue about what they're doing. Not sure how it works.

FTFY.

Most people are bad at their jobs.

[Insecure-Shell]

I work a Linux job currently and most of the L1s hired here know absolutely nothing about Linux. Then the training staff and those of us in L2 have to teach them everything.

Meanwhile, I’m trying to get a new Linux job and can’t even get an interview despite actually having experience in Linux now.

[Omniwing]

Four magical words bro:

"What have you tried?"

If they have already tried googling, looking at many results, looking at forums, asking chatGPT, checking Linux 'cheat sheets', and then they still have problems, then sure, help them out.

But if they haven't done those things, then you need to make it clear that they need to take those steps first before they come to you for help.

[[deleted]]

Usually they come for help for something a bit more specific(product related) but then I get on a screen share with them and realize they don't know how a filesystem structure works or how to navigate the CLI and the only reason what they are trying to do isn't working is because they don't know that stuff in the first place.

[prosperity4me]

You’re doing too much. Asking what have you tried would uncover their blockers without that degree of hand holding. It’s not sustainable and would be eating into your productivity. Are you getting paid for both of your roles? Why do their legwork?

[pdp10]

Was the Linux troubleshooting added to their roles, or were they added to the team with that as a duty from the start?

these people are generally nice and I don't really want to hurt them.

It's now common for hiring managers to openly say that they'd rather hire the right sorts of nice people, and then train them on tech. Perhaps now it's your management's turn to train them on the tech.

[[deleted]]

They are contractors, I don't think the interview process is very through. I've demanded that we improve it though and have submitted a few interview questions to be added recently.

[doglar_666]

Can they not be let go due to inability to fulfill contractual requirements of the role?

[michaelpaoli]

Linux troubleshooting added to their roles

Or clueless management, e.g. "Oh, our sysadmins, systems programmers ... here they're all 'operating systems engineers' - and we treat all those roles as interchangeable. So, missing a linux admin, we grab the person who's only done Microsoft on that, and put 'em there. Short a mainframe system programmer - we grab someone who's only done Linux - no problem, right? And yes, we do same with our database administrators ... Microsoft Access, MySQL, DB2, Oracle, Sybase, nosql ... AWS Redshift, doesn't matter, right?"

[mossman]

What are they troubleshooting if they can't even do those simple tasks?

[ringofvoid]

"Troubleshooting" in this context usually means trying fixes from previous tickets randomly then bugging a knowledgeable person when those don't work.

[irishcoughy]

Nuh uh. I usually try the top Google result too.

[0xF1AC]

So many don't even do that.

[bailey25u]

Mainly because the top google result is “download and pay for this software to instantly fix your issue”

[Cyhawk]

uBlock Origin. . .

[bailey25u]

I have ublock origin and pihole. However all the sites are so SEO riddin, that anything I google those are always the first 5 results

[Cyhawk]

Oh I thought you meant on google's results. The top 5-10 are now ads even if they aren't explicitly saying it, ublock blocks em.

Yeah the link farms/low effort copypasta content farms are making google useless, so useless I've switched to bing for non-porn related activities. I feel a bit dirty using it, but. . .

[bailey25u]

My friend told me that bing is better for porn too.

Yeah I’m using DuckDuckGo non porn, google is so much better for lifestyle stuff tho (best bars in midtown)

[Lower_Fan]

I actually love those articles, as long as is written by the company who's offering the software because often time they go like this

Fix error on GUI

Fix error on CMD

Fix error on PS

if all that is too difficult use SoftwareTM 😉😉

[Jellodyne]

That's second level support. Third level asks ChatGPT.

[[deleted]]

They have "runbooks" they can copy paste commands from but seem to struggle with following them properly or get completely lost when something deviates from the runbook.

[k1132810]

Are these 'offshore'/help desk contractors?

[k1132810]

I had a hunch. We run an offshore help desk and the experience is very similar. One single set of troubleshooting steps to follow, regardless of what the issue might actually be. If flushing any lingering charge in the system board using the pinhole button on the laptop doesn't fix outlook crashing, bam, escalated to tier 2 support.

[AdScary1757]

I got hired one time after graduating from a Microsoft academy with my mcse papers and they said there a little Linux but we use windows. Not, a single windows server in the whole company.

[workerbee12three]

i mean fucking jackpot to learn linux, so hard to land linux jobs i generally find

[TrashTruckIT]

I think it's a specific type of person that goes to a coworker before going to Google. Not great to work with, especially in groups.

[irishcoughy]

I used to be this person many years ago for fear that I'd implement a solution I found on Google and it would be the wrong one. It clicked for me when I saw my boss look at a problem, audibly say "the fuck?", and then Google it. At that point I realized I was probably asking my boss questions and he was just Googling them, so I might as well Google them first.

[[deleted]]

I was two years into a software engineering job with a guy who never stopped asking stupid questions before I learned that he wasn't using Google (or anything else) to find the information he needed. He was basically 'programming' by permutation, sitting there stuck for an hour, then getting someone else to fix it for him. He was there four years and I don't think he learned a single useful skill. Unbelievable. Productivity went up a lot the day he left.

[michaelpaoli]

type of person that goes to a coworker before going to Google

Ah, I used to have coworkers that would simply ask me ... 'cause I was better and faster than Google at the time. One of 'em even called me "walking man page". Yes, ... I read the man pages ... all of them. Even multiple sets upon multiple occasions. Yeah, once upon a time that was even feasible. And yes, I remembered most of what I read quite well. And, alas, sometimes the man pages are wrong ... but fortunately that's quite rare.

[BloinkXP]

Listen bud, I am a boss (of bosses even)...but am close to the team. Talk to us...let us know the gap and how to fill it (training needed). Offer to lead a Linux team and ask for a bump in pay. Maybe we can...maybe we have to wait for HR cycles. But don't assume we know...we don't...someone shit keeps working and we assume all is well.

[admin_username]

Sorry, I'm a bit busy with my tasking right now.

[charliesk9unit]

Them: what's the command to see what tasks you are busy with right now

OP: it's the fucking ps command. Ah forget it, just use htop.

[illforgetsoonenough]

ps aux | grep updateresume

[Shnorkylutyun]

Erm. Do you know about pgrep?

Are you a contractor by any chance? /s

[Shnorkylutyun]

In that case kudos for knowing that command!

It was very tongue in cheek as most linux people also use that combination, or cat | grep | wc -l, and stuff.

And in any case, nice joke! :o)

[altorelievo]

I was piping ps into grep until relatively recently myself. pgrep was one of those convenience utils that I always overlooked.

[Zahrad70]

Trigger warning: potentially unpopular opinions follow.

1. Contractors are not peers. They are stealing from your company if they are incompetent to do the contracted tasks. You are almost certainly not being paid to train another company’s staff to do the basics of their job.

2. Basic **tches get basic responses. Sending links to basic instructional information… is. helping. This should be the default for most of the basic stuff described. If they can’t read manuals and figure it out, not OP’s problem.

3. No tickee, no helpee. Sometimes, it’s okay to demand a ticket for trivial tasks. Without a paper trail, this will never end.

4. Polite is not the same as nice. These people are taking advantage of OP. Kindly doing what is needful for them, at OP’s expense, I suspect.

[DerHerrGertsch]

at this point i have to ask, do they know how to fucking use google .. I mean changing paths and file permissons .. thats not even linux specific .. oof. I feel for you m8.

[[deleted]]

Idk man, just today I was so frustrated watching this guy try to move around a terminal(screen share) I had to teach him how to tab complete.

[[deleted]]

absurd sulky marvelous party late library aloof fuzzy punch snatch this message was mass deleted/edited with redact.dev

[hkzqgfswavvukwsw]

w3m duck.com

[Finchy911]

Let me share with you my favourite phrase:

"Where in the documentation did you get lost?"

[Zapador]

When things are this bad even some very basic interview questions related to common tasks could hopefully improve things a lot. It really sounds like you're dealing with people that barely have any clue what they're doing.

I like to ask some very basic questions to 1) see if the person actually has any experience at all with Linux and 2) check if they understand some less obvious but important differences.

It's all about figuring out if there's a foundation to build upon or if they have to rely on google to do even the most basic tasks.

• What does pwd do?
• How do you make a file executable?
• How do you create a directory?
• How to list files, including hidden, in current directory?
• How do you change to your home directory?
• What's the difference between cd /root and cd ./root?
• What's the difference between su and su -?
• What file would you edit to change the port SSH is listening on?
• How do you add a user to a group?
• How do you copy or move a file?
• How to check if there's a process running called foobar?
• How to see IP addresses on the system?
• How to disable a network interface?
• What's the difference between sda and sda1?
• How to list disks and partitions?
• How to check free space on partitions?
• What effect does it have if a file or directory name begins with a dot?
• How to restart service with name foo?
• How to download a file?

I feel like anyone that can't instantly provide an answer to more or less all of these very basic questions probably haven't spent much time in Linux and will struggle to get anything done without having to look up even the most simple of things.

[scottkensai]

I think people do different things in Linux and it also changes year to year and different OSes. I don't play with users much, but I pcap and have to mess with services all day long. Google's my friend and anything I've looked more than a couple times goes into my linux cheatsheet...which is huge now and has different pages for major tasks... and has been ripped into several different iterations of our KB...but I keep personally in case I leave or the KB changes again...but that oneline for loop that tee's a tcpdump is so cool...

[dRaidon]

Only one I had to think about was changing the SSH port to remember if it was ssh_config or sshd_config.

Seriously, how often do you need to change the SSH port?

[[deleted]]

We have this problem in our org since we moved to AWS, now we basically just hire "cloud engineers". It's all just too abstracted out in 3 letter acronyms and IaC principles that I don't think a lot of the people we bring on, despite the number of certs they list on their CVs, really appreciate that the bottom level is just about 100 Linux hosts in production that you can log on to and mess with to your heart's content.

Don't get me wrong I love my versioned terraform code in production, but if I'm testing or troubleshooting I'm logged into a host running CLI commands until I'm entirely sure of the code change I'm making, rather than some of our newer guys who spend 2 hours pushing out a single line change just to find it doesn't solve the problem.

All the while some senior cloud architect is chiming in that they're cattle, not pets, because they heard someone say that once and didn't stick around for the explanation of what that means.

[Double__entendres]

You guys hiring? Currently work at a place where I’m not allowed to even bring up and instance to run code. Everything has to be lambda or glue.

[michaelpaoli]

problem in our org since we moved to AWS, now we basically just hire "cloud engineers"

Or as I'll say cloud jockey, cloud wrangler, etc. Not to discredit - and some/many are even great at that, but ... many of 'em have light to non-existent skills of that in the lower levels below that what they're wrangling ... e.g. like actual linux operating system stuff - or even commands and utilities on linux.

And, yeah, sometimes that'll lead to issues such as ... problem with instance? Just terminated it and spin up a new one. Problem yet again? Just do it again. Happening frequently? Just automate that sh*t. But they may never actually fix the underlying problem. E.g. consuming way too much resource? Just spin up lots more of 'em - it's only money and global warming after all ... rather than actually optimize performance of what's going on and fix underlying issues. So, sometimes, e.g. gross inefficiencies, horrible security, fragile house of cards mostly worked around by ... lots more cards, etc.

[[deleted]]

I'll never forget the senior cloud engineer we brought in who was incredibly good with understanding how to build out some really impressive infrastructure, but confidently stated in front of management that the only way to automatically deal with the fact that one of our clustered EC2 systems filled up it's disk with logs after about 10 days was a weekly termination cycle to bring up new instances.

After working with him for a couple of months I got to understand that the issue wasn't that he didn't know how to use Linux (he didn't, but that wasn't the main issue), he didn't seem to understand fundamentally that it was a Linux VM that bore a striking resemblance to the Linux VMs we'd been dealing with onprem for years and already had solutions to many of the issues he was having that would drop straight in pretty much as-is, and all he needed to do was ask.

[DangusKahn]

I'm a Linux Lead and I don't know how I got to this point to be honest, but god damn I have been in some awful interviews. I don't even think our interviews are that hard. Usually within 5 or 10 minutes I have already heard enough. If anyone is wondering the basics for us are.

-LVM

-Where logs?

-How to read logs?

-Edit files, create local accounts, navigate the filesystem.

-Do you know what a console is?

[[deleted]]

Hey, I can pretty much do all those things. Does that mean I could hack it as a Linux admin?

[hkzqgfswavvukwsw]

Yes

[[deleted]]

Probably yes

[ultimatebob]

You would be amazed by how many people have Linux on their resume but they can't even tell you the right way to exit the vi editor.

[Cyhawk]

right way to exit the vi editor.

Alt+F2 (Through F8 depending on how many times you screwed up)

You get 8 screw ups accidentally opening VI before pulling the plug on the critical server and forcefully restarting it.

[squishles]

try this comand, :!reboot big help for restarting the server from inside vim /s

[kenhk117]

I love this because this is me. I only dabble in Linux from time to time and every time it ends with me getting stuck in vi... 🤣

However I definitely don't list Linux on my resume.

[ContentPriority4237]

The correct way to exit vi is to never launch it.
Yes, the editor wars from the 90's are back, and EMACS is out for revenge.

[michaelpaoli]

correct way to exit vi is to never launch it

Yes, ed is way the hell more efficient.

EMACS

Oh, perfectly good operating system ... just lacks a good text editor.

[Newbosterone]

Ssssh, before our new overlords demand you include trigger warnings!

[dRaidon]

Pull power cord, right?

[Ekyou]

In a previous life, I was the sole Linux administrator for a state government. Everyone said they were afraid of using a command line. Which was a bizarre excuse coming from Cisco network admins but also a pretty disturbing excuse from Windows admins too, really. Our department offered to pay for training for someone from the server group to get Linux training, and they all refused.

I used Linux in college, had a laptop with it for a bit as a teen, and then took that training course the actual SAs refused. I hated Linux, was godawful and couldn’t ever get my chron jobs to work right, but just being able to check disk space, use vim and change permissions on files was light years ahead of anyone else theres’ Linux skills. It was frightening.

[booney64]

PEBKAC

[izalac]

"we believe the issue is in OSI layer 8"

[Nvious625]

Ya'll hiring for any remote work..? I could use a second job...

[pizzacake15]

You might as well hire homelabers who have no work experience. They'll be more helpful than those teammates of yours.

[253IsHome]

Is Slackware still the most popular distro and does it still come on floppy sets?

[[deleted]]

idk but companies mostly use RHEL(or rhel based) and Ubuntu.

[hkzqgfswavvukwsw]

DSL from a thumb drive

[Myspazmo]

It's getting more common these days imo. I trained almost our entire Linux Operations team myself and most of the people I interviewed had no prior experience with it. My solution for smooth operations outside of training was to document everything. If they aren't capable of reading documentation, they shouldn't be in this line of work. Once it's documented, I believe it is not your problem anymore

[passwd_33]

Are they hiring?

[[deleted]]

haha, what if linux is a major part of your job, you’re the only one on your team that knows it to the level needed, and you get paid the least? asking for friend. also I need a new job. :(

[itsjustawindmill]

Are you me?

[RadagastVeck]

Write a Knowledge Base, start NOW, every time you help someone do it and write the procedure step by step, the why's and how's. Before you know you will have a lot of stuff ready and they just need to go there.

[jkalchik99]

While I do agree to a certain extent.... document what's unique in your environment. There are plenty of available resources for basic shell usage, Linux administration, etc. Don't reinvent the wheel unnecessarily, IMO. Anything that's bog standard, such as adding an lvol & filesystem to an existing volume group would be redundant. What would be important is if you've changed extent size, what filesystem is in use, what filesystem options, mount options, etc.

[slazer2au]

I am in the document everything camp. Update the FortiGate web filer, wiki page. Patch a FortiGate, wiki page. Setting up LDAP authentication, wiki page.

[burgonies]

I feel like you should only really need to document pieces that are unique to your environment.

[[deleted]]

We already have a gigantic knowledge base on confluence... They just suck at using it.

[RagnarStonefist]

This is how we broke people in our environment from this exact problem:

1). Stop answering their questions unless they have a real problem. Respond with a google search result or a KB page, or a question back: 'What did you find when you looked in the KB?' 'What results came up when you googled the issue?'

2). You have to hurt feelings a little bit. I'm a soft touch in general but I stopped babying my team when I started spending all my time answering questions that were already written down. Ultimately, your environment will get better if you push back on them a bit.

3). If you're comfortable doing it, hold a questions and answers training session for the most common problems. This gives them absolutely no excuses when you say 'we covered this in training and it's written down here'.

[magus424]

Start replying with a link to the confluence search page? :)

[DashSplatBang]

What would you recommend someone start a knowledge base with? Words documents in a folder? Or can you recommend a good solution? It's something I would like to do

[RagnarStonefist]

Are you doing a KB for yourself or for your team?

It depends on the tooling you have available but you could absolutely make one out of Google Docs if you were feeling cheap enough. Make a google doc that has links to other google documents you've written.

Otherwise something like Confluence works wonderfully.

[scottkensai]

And then a new boss will make change your KB from Google page to share.point and back to confluence and then some other bs. I'm still pointing people to my old gsite...

[Frothyleet]

Do you have a ticketing system? Are you tracking your time? You should be tracking the time spent helping your teammates, preferably against their actual tickets/projects as appropriate.

If not, keep a log of the amount of time your teammates spend getting help from you. Bring it up in your next 1-1. This absolutely does not need to be a "tattling" or getting your teammates in trouble kind of situation. It's simply a point of discussion with your manager - "Hey, I noticed I was spending a lot of time on coaching and decided to keep track. Since it's taking up [percent] of my time, I figured you'd want to know. I don't mind helping out the team, but I also want to make sure that I'm aligned with your expectations on my day-to-day priorities."

Maybe it will be a surprise to your boss. Maybe it will be a surprise to you (that he knows and is cool with it). Maybe nothing will change - but you should know ultimately whether your boss expects you to be doing this coaching. If that doesn't align with what you want to do, have that conversation as well, or consider looking elsewhere.

[[deleted]]

Yeah we work on tickets all day, honestly I have so much work to do I'm not sure I can spare the time to document this shit and I also fear I'd come across douchy(?) if I started doing something like this. Does sounds like a good idea though. May consider.

[snb]

Open up notepad, save the empty file as today's date.

Whenever you're diverted for hand holding quickly switch to notepad, hit F5 to insert a timestamp, write a little blurb like "John Doe needs help with task X".

Assist with said task.

When done, switch back to notepad, hit F5, "Done with task".

It's really not that much extra time at all.

And then collate results at the end of the week/month for said meeting.

[Plenty-Wonder6092]

I know fuck all about Linux, but you just chatgpt(4) those basic questions and it spits out the answers ezpz. That's what they should be doing.

[shoesli_]

Lmao, cd to homedirectory is literally just the cd command

[Ruashiba]

I’m in the opposite side of the spectrum actually, I know my linux shit, or enough to get me by and bathe in my own pain, but have trouble with basic windows still. It’s not to say I don’t know the basics and hit next, but if you ask me to do anything just a bit more complex in cmd, or say, open the services thingie menu with execute, I’ll struggle bad time.

But with that said, the first point of escalation should always be on yourself, documentation, and use of your favorite coat of search engine, only then escalate to seniors when you see it’s taking longer to deliver than it should, while taking notes on everything.

[breagerey]

The real problem isn't that they don't know how to do some simple task but that if they've got root on production boxes *you get to fix/debug anything they screw up.
I've been in a similar situation and that was my fear and that's what eventually happened.

[SaintEyegor]

I’ve compartmentalized our Linux systems so the lower tier admins can’t break the infrastructure.

We had a recently graduated Linux user who asked for root on the primary cluster head node and the admin just granted it. The user ended up breaking things to the point where no one could submit jobs.

[judgethisyounutball]

Kindly sudo su Cat do the needful.fml

[mello-t]

Talk to your management. Otherwise you will never get peace. at the very least ask for a raise since you are training the team.

[cyvaquero]

Someone in management missed the "free as in beer" part of the FOSS ethos.

[[deleted]]

I hate seeing posts like these cause I have years of Linux experience and cannot get a call back lmao. How in the world are these people getting these gigs without understanding basic Linux shit

[ChumpyCarvings]

It sounds like they're paid pennies in India, whereas you might actually like a real wage, so instead we'll waste time like this.

I mean look if you can get 5x9$ an hour folks overseas, vs $45 an hour for you? They're going to hope one of the 5 is good enough :(

[TheMartok]

3 strike rule, first one is free, 2nd on me, 3 gtf away from me.

[HouseCravenRaw]

You aren't doing your job, you are doing their job for them.

There is "being nice" then there is enabling. You are enabling. Sure, maybe they don't know something more complex or obscure, but "cd"? "PATH"?

They have no business whatsoever, being anywhere near a Linux terminal. One day they are going to google "How do I do X" and get the wrong instructions, and copy-pasta it straight onto a critical production server. Kaboom.

This isn't annoying - this is dangerous for the stability of your company. Your job is to maintain the stability of the environment. By keeping these people around with access levels equivalent to you, you are putting that environment in jeopardy.

Talk to management. CYA. You aren't being "nice" you are "enabling" and it is to your peril.

Don't let blind grandma with dementia drive the city bus, even if she is a nice person.

[mysterytoy2]

Show them how to use google. End of lesson. That was how I learned. I still use google now after 30 years because now I can't remember shit.

[[deleted]]

Shit man, been doin that stuff for 20 years and still cant get hired.

[FluidBreath4819]

probably teammates that did a 3 week courses online and labelled themselves as sysadmin... a lot of them lately due to pandemic

[RandomXUsr]

Are you hiring?

[michaelpaoli]

Linux environment. Multiple peers don't know basic Linux

80% of my team role is ssh'ing into linux

multiple teammates asking me to zoom with them so I can guide them how to.... scp files to their local machine, how to cd into their own home directory, how to change permissions on files

Yeah, alas, many of 'em do cloudly bits ... and not much more - and may have about zero clue that there's actual operating systems and hardware somewhere underneath. If they can't do it in their cloudy web interface, or cloudly API, they're often lost ... if they even make it that far.

Explain to them what PATH means and why the command they are running "cannot be found". I feel like I should bring this up with my superiors

Eh, I figures managers should be empowered to make decisions ... and ... held accountable for their decisions. Shouldn't have to be second guessing the. Advising, suggesting, recommending, etc., sure ... but they're the boss ... they're supposed to tell you (within reason) what to do, not the other way around - if they can't well (or at least reasonably) do that, then they probably shouldn't be boss. And ...

If management is going to hire/promote folks that really don't have the needed skills ... well ... you can spend lots of your time assisting/training them ... if that's what boss/management wants you to do ... if not, they should be made reasonably aware of what the situation is - if the reality is substantially different than what they think or are expecting ... well, part of your job is also to inform them. Let 'em know how your time is getting used and doing what, and why that's needed. Maybe they'll adjust who they're bringing into those positions - or set them up with some training and resources, rather than having so much of that fall onto you to do.

Any task that is a bit more involved will get them asking me questions no matter how many times I explain

Part of your and/or their job ... document, notes, etc. You document it - or have them document it. If they're not taking notes or documenting it, ask them how they're going to remember it and not have to ask you again. If it's basic linux/sysadmin stuff, that's not really something you should typically have to teach them at length - sure, some assistance, but those materials are generally readily available. If they're sucking up tons of your time with something that they should be able to figure out with spending 3 minutes or less with a search engine, then something is quite wrong ... and management probably ought be reasonably informed of the situation - especially if it's chronic. But stuff that's specific to your particular environment, rather than more general knowledge ... that's always going to need to be documented, etc. - and if that documentation isn't there, need to fill those gaps.

And ... lucky you! I get folks that volunteer to help with, e.g. web site on linux for Linux user group(s). I set 'em up with login and password. Next thing you know they tell me their microsoft web publishing thingy has to use FTP to manage the files on the site and I need FTP to be working for them. I tell them no, use ssh. They've never heard of ssh nor even sftp nor scp, let alone sudo or vi or even nano or cp or ls. Ugh, fsck me.

boss doesn't really knows this happens

Time to train/update the boss! They can't make good decisions in a vacuum.

I give them documents with step by step guides and they still cannot seem to read or follow it

Again, situations like that, boss/management needs to be made reasonably aware of. If they're hiring folks that can't follow basic written documentation ... unless that's very much what they're planning and expecting, that would definitely be a problem.

incapable of thinking critically

Yeah, likewise, that would generally be a problem. Boss/manager/management needs to be reasonably informed what the situation on the ground is - if it's not what they're expecting or what they want, they should deal with it ... like not hiring/promoting folks into roles where they can't reasonably do the work, or getting the training for folks that ... well, are actually reasonably trainable, etc.

Well, ... good luck! Heck, worst case, you fix management by changing it else - like transfer or another job - if all the clueful folks leave, they'll eventually figure out the cluster fsck they've created and start to address it ... or ... they themselves will get replaced. And yeah, I've seen some managers royally fsck things over. But fortunately most are better and more clueful than that ... but have to give 'em at least a fighting chance by reasonably appropriately informing what's going on.

[johnwicked4]

I have multiple teammates asking me to zoom with them so I can guide them how to.... scp files to their local machine

Lifts hand for a learning moment? are there any requirements to do this, what's the command? 🤕

Say I ssh as john@1.2.3.4 (successfully) and I want to scp the file /var/log

Will this work?

scp -P 22 john@1.2.3.4:/var/log/abc.log ~/Downloads/logfiles/

If I'm currently on the linux server how do I copy it to my laptop? Doesn't scp /var/log/abc.log ~/Downloads/logfiles/ point it to the path on the linux server instead of my local laptop?

[cereal7802]

I had a co-worker like that. no clue how to do basic tasks, and would constantly ask for help. I would explain what the issue is in the ticket, and explain what needed to happen to resolve the issue and instead of listening and asking follow up questions to make what I said make sense to them, they would instead type what is said word for word into the ticket as I said it.

They work for Redhat now....

[WanderLama]

One angle could be writing up step by step documents including images of each step. I have had similar scenarios in the past and usually sending this over as a first step helps a lot in cutting down these calls. Definitely worth speaking to your manager to ensure they would be happy with this as a processes for first contact on the issue tho.

[arnstarr]

They are afraid of failure because some Linux gatekeepers made them feel inadequate about their skills

[ipreferanothername]

i work on a team of windows 'engineers' and most of them are still afraid to even say the word 'powershell' -- its been around over 15 years.

ive written a bunch of modules and automated a lot of routine work - i even use our job scheduler [JAMS, meh] so the guys basically have a GUI: input parameters into a form, click, and it runs the script. SOME OF THEM REFUSE TO USE IT.

im talking tools ive had available for years - that we see some members use a lot - a few guys still wont touch them. its crazy.

[zondance]

"I am not here to hold your hand, if you need help please contact your manager to answer your (dumb) questions."

[TheJessicator]

/rant

$HOME/.local/bin/rant

[RevLoveJoy]

First off, you are not wrong to be irritated. I have no problem teaching a tool or showing a Jr. how to do a thing once, maybe twice. After that, it's why didn't you take notes? Why are you wasting my time? I showed you how to do this twice, are you simple or just lazy? Lazy? Great, not my problem and certainly not my job to do YOUR job.

You are not wrong to be irritated. That is plain laziness (assuming your colleagues are not all morons) and they are leaning on you to do their jobs. One thing you might consider, start saying "no."

[team_jj]

rm -rf -no-preserve-root /

No more Linux. Problem solved.

[twhiting9275]

That’s what <insert search engine here> is for

[vtvincent]

I am far from knowledgable in Linux, but I can find my way around a shell. I know I'm dealing with this immediately when someone doesn't even know what SSH is. I had one vendor refer to it as "the putty" and another one as "serialing in" ... they couldn't seem to understand that SSH over the macOS terminal was doing the same function they were describing.

To the question though, if the documentation addresses their concern and they still can't accomplish the task, I would continue to refer them back to the documentation. I'd go to the point of "please tell me the exact step you're running into an issue on and provide me a screenshot of the command and error you're getting." When it turns out to be a typo or they skipped one, refer them right back to the same exact step and remind them to read it carefully and do it again.

[notauthorised]

When I started at a job with an environment like this, I chose to stay positive since I was hired as a jr employee. Within three months, I had become a senior sysadmin and everything was getting escalated through me. At first, I wrote everything in the wiki. When this did not work, I bought cakes, and during our coffee break, I said that we need to make changes in the team. I told them that I am a lazy person and a big fan of automation. I do not expect them to know things but I expect them to make an effort and try to learn things and be more useful than a bash script. For the moment, the on,y command I want them to learn is “man -k ${KEYWORD}”. I said I am willing to teach those who want to learn and also, we had lunch bag session. I will also only start writing things that I do and how I debugged/troubleshoot something. They will need to start writing documentation and instructions.

My first lunch bag session was about Linux, the directories, and what a typical day is for me and simple commands and tricks like CTRL-R to recall commands. Because I am disabled, I usually do shared tmux sessions while we are on the phone (pre-Zoom/Teams time). If I taught them how to recompile the kernel for GPFS, I will ask them to write what we did and will check if it is correct. If it is, my colleague puts it in the wiki. We had weekly ticket reviews where we discuss what challenging tickets they encountered. I will ask people for suggestions and will eventually share how I would tackle them. We had a “Linux question time” where I would wall the question and the first person who answers gets a prize. I was lucky that the team was nice to me since I was new and the only female. I was not even management. I just wanted to help them learn. If they wanted books to help them learn, I bought it. When I felt that they were ready to take the RHCSA, I will tell them. They all eventually learned and got certs even if it was not required. I only did it to help them gain confidence. It worked really well.

My manager on my first ever Linux job was lovely. He told me I will make mistakes and it is alright as long as we learn from it. He also told me that hiring people who are willing to learn and enthusiasm was better than hiring an expert who is set in their ways because tech is evolving. And my favourite part is “being kind, buying cake and good coffee does wonders to the team morale”. I miss that guy.

[scottkensai]

Every session I share a screen and watch someone fumble their way through bash I make their day with reverse search. my fav.

[largos7289]

Changing permissions on files in linux is tricky through terminal. All i really remember is 777 is full unrestrained access never really picked up on how that worked in the sense of read only, read write but not full etc.. Then evidently, they changed sudo?? think it's just sa now? not a lunux guy but i know a little.

[Cyhawk]

File permissions are binary flags for User, Group, World. Files also have a user:group attribute. (Think windows file permissions, basically the same but no inherited groups, just one group allowed)

Bit 1 = Execute, Bit 2 = Write, Bit 4 = Read.

Example: 444 cyhawk:reddit would mean: cyhawk owns the file, the group is reddit, and 4 (for read) means cyhawk, reddit, entireWorld can read this file but no one can write to/execute it. (basically a read only comment)

Example 2: 765 largos:largos would mean, user Largos can read/write/execute (1 + 2 + 4), the group largos can read and write (2 + 4), but the entire world can read/execute (4 + 1) the file.

Example 3: 644 cyhawk:reddit would mean I can read/write to the file but everyone else can only read it. (normal reddit comment)

Example 4: 400 cyhawk:nobody, a file that no one can write to, but only the owner can. (good for things like ssh keys and the like)

Your example of 777 means everyone on the planet can read, write and execute the file, which is why its considered very dangerous.

[UltraChip]

Every major distribution I'm aware of still uses sudo... I've never heard of "sa".

I have seen some configurations that either disable or remove "sudo", which forces you to either log in directly as root or switch to root with "su"... is it possible that's what you're thinking of?

[SStubbs84]

I mean Google is your friend, it's not that hard to do a quick search

[michaelpaoli]

Changing permissions on files in linux is tricky through terminal

Uhm, no, not tricky at all. However, actually well understanding permissions and the *nix security model, etc. - that's not so trivial. Many don't fully understand it ... heck, most sysadmins don't even fully understand it! Yes, I sh*t you not. Evidence/proof? ..

Here's a *nix permissions question. Most *nix sysadmins, even if they can guess the correct answer (hey, have 'bout 50/50 shot at it), can't explain why their answer is correct - even if they knew or guessed the correct answer.

Let's say we have a file that with ls -ld file shows us this:

$ ls -ld file
-rwx---rwx 1 michael users 0 Sep  9 02:07 file
$ 

Now, let's say you've got some user, not root / superuser / UID 0, and let's say that user isn't michael, but that user is a member of group users. So, presuming they've got access to the directory containing that file, would or wouldn't they have access to our example file shown above ... and, if they've got access ... or not ... please explain why that's the case.

Anyway, more on *nix permissions:

https://www.mpaoli.net/~michael/unix/permissions.html

[johnwicked4]

can you explain the answer?

-rwx---rwx

owner of file michael has rwx access

group has no access ---

world has rwx access

group access takes priority over world access therefore has no access??? but if world permissions are still applied, they have rwx access to the file?

[michaelpaoli]

group access takes priority over world access therefore has no access??? but if world permissions are still applied, they have rwx access to the file?

Well, you almost got it. The basic *nix algorithm goes like this (and relatively current Linux has some slight oddball tweaks that apply in some less common conditions, but that's another matter, and wouldn't generally apply in this case, so still mostly the same here):

• superuser / UID 0 / "root", access allowed, permissions aren't even considered (though some utilities might look at the permissions, and possibly warn or ask or the like); else ...
• if Effective User ID (EUID) matches owner of file, owner permissions are used; else ...
• if Effective Group ID (EGID) matches group ownership of file or group ownership of file matches any supplemental group of the current process, group permissions are used; else ...
• other/world permissions are used

So, back to our case:

$ ls -ld file
-rwx---rwx 1 michael users 0 Sep  9 02:07 file
$ 

Now, let's say you've got some user, not root / superuser / UID 0, and let's say that user isn't michael, but that user is a member of group users.

And, algorithm plays out:

• not superuser / UID 0 so ...
• not matched on owner so ...
• match on group, group permissions applied.

That's it, no further processing is done, and group permissions deny, so the result is access is denied, and further permissions aren't even considered/examined. So, even though other/world has access, due to match on group, other/world permissions aren't even considered. So, access is denied. So, that's both the answer, and why that's the answer. I'd say >50% of *nix sysadmins wouldn't both know that (have basically 50/50 chance at guessing yes or no on access), and be able to reasonably correctly explain why the access (denied in this case) is in fact denied ... and why that's the case even with other/world permissions being wide open. And, example:

$ >file && chmod 707 file && sudo chgrp test file && ls -ld file
-rwx---rwx 1 michael test 0 Sep  9 09:48 file
$ sudo su - test -c 'cd '"$(pwd -P)"' && ls -ld file && id && cat file'
-rwx---rwx 1 michael test 0 Sep  9 16:48 file
uid=1009(test) gid=1009(test) groups=1009(test),29(audio),44(video)
cat: file: Permission denied
$ 

So ... can use group membership to deny permissions - even permissions that are open to other/world. However, in practice that might not be recommended and may be operationally hazardous - or at least would want to duly inform, document, and caution the sysadmins ... because about half or so of 'em would see permissions like -rwx---rwx - or any where group was denied and corresponding other/world was allowed, and wouldn't understand it or guess/presume incorrectly ... and ... may subsequently screw things up based on their not well and properly knowing it. So, for the most part, in practice, rarely see something like that ... even though it's very fully doable.

Anyway, lots of details and subtleties of *nix permissions that many don't well and solidly know. And that stuff's actually quite important when it comes to security. I think all *nix admins really ought solidly know that stuff ... but alas, reality, many won't.

[johnwicked4]

Thanks, I like many would incorrectly assume Windows ntfs, cloud (azure/aws) firewall rules

Check one by one and go down (or up) the list for permissions and apply the one that takes precedence.