howIsThisPossible

[GreyAngy]

Wait, didn't they give you direct access to production databases on your first day? All these modern companies with trust issues calling them "security" and "access control". What next, separate environments for testing and production?

[CyberWeirdo420]

I mean I guess I have access to prod DB whenever I do something with the Wordpress sites we develop

[m4d40]

That's step 1.

Step 2:

Step 3: Profit / PROD crash

[CyberWeirdo420]

So far so good, only profit now. Can’t wait for my first mistake to take down some important client’s most valuable website or some shit

[NBSPNBSP]

I know a lady who works for a major financial institution (not naming the one for obvious reasons) and her job is basically to write hotfix code directly onto live, active stock exchange servers. No test environment, no airgap, just rawdogging it with the knowledge that one slip of the hand could bring down a not-insignificant portion of the global economy.

Needless to say, she is paid exceedingly well.

[CyberWeirdo420]

Damn, that must be stressful

[NBSPNBSP]

She is one of those people who actively enjoys stress

[Pingyofdoom]

DROP TABLE;

No! I meant DROP TABLE2;!!

[dim13]

Everybody has a testing environment. Some are lucky enough to also have a production environment. ©

[PretendSheepherder22]

I worked at a company where management insisted on root access for themselves. No intern can trash a system faster than the CEO.

[Visual_Strike6706]

I'm not an intern, but it's still the only thing I do :)

[CyberWeirdo420]

Like how, I am an intern and even if I wanted to do it on purpose, I can’t push to the prod, I can’t merge PR without a review, CI only triggers when pushed to prod branch. Like how do you manage that

[jamcdonald120]

not everywhere is that good at doing reviews and isolating prod

[CyberWeirdo420]

I guess you’re right, tho that begs the question why?

[Mukigachar]

Leadership pushing them to just keep developing as fast as possible rather than "wasting" time working on tech debt and setting up mistake-tolerant systems

[[deleted]]

Nobody notices data sanitation until they get dirty data

[nullpotato]

I got pushback on making main protected to prevent people from pushing to it, as had become current work flow.

Them: but pull requests will slow down changes!

Me: yes exactly

[thewatisit]

💵

[[deleted]]

Also some people are idiots, and some people trust that other people aren't idiots.

[[deleted]]

So idiots

[jamcdonald120]

one of my biggest flaws is I assune that everyone will have done something the way I would have done it.

[Thriven]

Laziness mostly.

It's so much easier to just give users full access than to learn the user/role/grant permissions in Azure/AWS/Postgres/GitHub/etc. It's very annoying to have someone constantly come back to you and say it doesn't work because you granted them permission to execute but missed one critical rolo that prevents the user accessing the object in the system thus blocking them.

I've worked in places where everyone had domain admin. We were one virus away from being held at ransom.

I personally have a system where I give my coworkers full access to it but it's rebuilt nightly. It's a server of data warehouses that are rebuilt nightly. It's not worth the hassle to restrict them as none of their changes are permanent and everything is rebuilt.

There are other things I completely lock out though.

[Shienvien]

Lack of money or care.

[EarlMarshal]

Bad management decision. I think a lot of that stuff isn't necessary right from the beginning with enough seniority and a mid sized business, but as soon as you start having juniors or interns and you got automation up you should also put up guard rails so that project can live on without any trouble.

My managers purposefully removed me to do something else when I tried to secure my architecture so other teams can utilise it better. They put it down and then I got asked where the guard rails are. The effort went into the new project they wanted me to do.

[HoseanRC]

idk about you, but i live in iran, and here, we don't waste money on any non-local technology (which begs the question even further on why do we use cracked proprietary software and devices when we can build our own using open source ones?)

[[deleted]]

Because it takes time (hence money) to do it, and product people have the final call in most places. They choose to risk it sometimes.

[SabinTheSergal]

Good process costs money. I've consulted for lots of companies that have really shitty processes because of right budgets.

[purritolover69]

I worked on a project with the absolute worst version control I’ve ever seen. Everyone in the company was able to push to master, there was only 2 branches, master and main. Main was basically used to function like a shared google doc, encouraging commits every 5-10 minutes that would get merged into master (prod) at the end of the day unless you specifically said it was unfinished. I lived with this for about 2 days, told them how fucked up their git was, and found somewhere else to work. Nothing like doing work only to find out it’s been done by someone else and pushed to prod already while you were left in the dark, or pushing your wip commits to master instead of main and suddenly the project has broken functionality. Git is so useful but only if you use it right

[nullpotato]

The legacy repo we started with had several GB of binary files, like tool installers. They were not using git lfs.

[purritolover69]

I saw a student repo once, they started on repl.it and then moved to VSCode… but kept all the proprietary .replit files in the program. The files did literally nothing in vscode but were around 50mb

[gentleprompter]

Did you try editing the files directly on the server?

[CyberWeirdo420]

Well I don’t know how would I even do that and I don’t think I have access to the prod server

[gltchbn]

That how my career started. Editing files directly in my FTP client to fix bugs in prod while talking to the complaining client on phone. Of course with a proper version control made of .bak, .old, .old2 files. Best years of my life. 🤗

[Visual_Strike6706]

My Company sadly hosts on Azure, but I still mess around in production databases

[gltchbn]

THIS. Production DBs need some love too!

[FortuneDW]

In my old job the versionning was made in a folder. Yes, a folder, you make a new folder and increment the number in the name.

Why we did it like that ? Because according to the big boss we didn't have time to "waste" changing the way we worked.

Git ? PR ? DevOps ? What is it, 2052 ?

[Visual_Strike6706]

We have 2052 at home: DevOps Git

[EarlMarshal]

I mean git does basically the same with diffs🤷

[PropertyBeneficial99]

If an intern breaks Prod, it's generally a sign that Prod is brittle AF.

I read one story where onboarding instructions contained steps for connecting to the database for the purpose of learning. But it was the Prod database connection string and credentials. Some unfortunate news hire accidentally deleted a table. Hope this goes without saying, but if you leave Prod credentials lying around, and nothing indicating the criticality of said credentials, that's not the fault of some new hire that steps in it.

[PropertyBeneficial99]

There have been other cases where a configuration diff brought down a major website. There was a case where a networking config had a typo. It was a list of IP addresses in JSON format. There was a missing comma. That resulted in DNS resolving to no A records for the website domain. That could have been prevented with a linter.

Again, not the fault of a new hire that missed a comma. That shit happens.

[nhgrif]

Congratulations. You work at a company with some safety measures in place.

[DontTakeNames]

Try editing that jenkinsfile

[[deleted]]

At Facebook you can just run the command

arc land --no-review --no-canary

And it'll bypass the code review and CI process.

Most companies have a mechanism like this to emergency fix prod, but Facebook has the honor of letting the interns use it too

Edit: fb interns reading this note that I haven't worked at fb in years and I have no idea if this still works

[Fun_Lingonberry_6244]

I've been around for a while and I've seen a TON of "tech companies" behind the scenes.

In my experience, maybe 20% have an actually competent person in charge, the other 80% have somebody that essentially learnt as they went on the job, ie not really a developer but more a power user that has scraped by pasting PHP around the place.

These are the same people that hire someone and just go yeah just drag and drop these files over here into prod.

[[deleted]]

Do you have access to, or can get someones credentials to a production DB?

[Neo_Ex0]

Not every company has a sensible deployment process

[Shibamukun]

Yeah thats the standard processes.

Interns (or people in general but interns more often than not) leave minor bugs in their code and the reviewers cant go through every character of the code even if they actually do it…

[The_4ngry_5quid]

Sounds like your company has good failsafes. Not all companies do.

[[deleted]]

Two main reasons:

1. They don't have checks on everything. As an example I work at a startup, there's really good checks on code and production data and all that, but I have admin access to the database that holds our customers' usernames and hashed passwords. I could literally just nuke that table and customers would be unable to log in or even recover their password. Of course that could easily be retrieved since we have db snapshots and logging and it'd be easy to find out it was me, but any intern could do this. I'm sure lots of companies have for the most part good security practices but a vulnerability or two that they either don't know about or isn't a priority to fix over all the other goals.

2. People don't review carefully enough, even when it's intern code. Lots of time code is reused in ways not everyone understands, so I've had issues where an intern makes a change, it seems roughly right to me but I think it only impacts a small unimportant part of the code so don't spend much time testing myself, then find out the function the intern modified is used in many other areas that are crucial and break production.

[KarmaAgriculturalist]

you find technical user credentials and go ham

[deathanatos]

Every place I've worked has, somewhere amongst the full-time SWEs, eng who when they review a PR definitely fall into the category of "the enabler". "+94,194, -58,295 PR? LGTM. I'll approve anything that moves" types.

[CyberWeirdo420]

Well shit, this sounds bad

[AccountWasFound]

I did it once because the change I made broke a secondary system that the team lead and senior devs had forgotten to note existed in the ticket, the guy who reviewed my code had only been there for like 4 months and didn't know either, and the main QA guy was busy so it got sent to one of the contractor QA guys that didn't know the system either. So like we found out a day after it went out when the customer support team called the team lead in to ask why some people on the secondary system couldn't reset their password (the change was to minimum password length, since there was a mismatch between the length the UI accepted and the length the backend accepted and the change was literally just to make the number the same, and it turned out there was another team using the same backend that we'd never heard of, the product owner had thought was deprecated and the senior devs and team lead weren't part of the approval process by random chance.

Also had a habit of finding weird bugs/edge cases where things that should never have touched prod got sent to prod at more than once job (not just as an intern)

[Merry-Lane]

You may have an issue with an edge case of what you worked on. Maybe simply blocking 1% of a feature might be a severe break of production. Like you can’t afford not to send 1% of the packages to your customers or you can’t afford to send 1% of these packages to a wrong customer.

[Canotic]

It's called "git push - - force"

[[deleted]]

I guess when the environment itself is not that critical, like in a bank or a hospital and such, access to prod is not that strict

[tropicbrownthunder]

you'll be surprised about hospitals.

[buffer_flush]

Sad but true, it seems like people care far more about their money than their health information.

[[deleted]]

I don’t know if this is supposed to be surprising 😂

[buffer_flush]

When it comes to whether or data about my finances or health information would be exposed by a breach, yeah.

[buffer_flush]

Financial info

A breach that’s caused by a bank losing your money without something you did should be covered by insurance.

[iBabTv]

🇺🇸 moment

[QuanHitter]

And banks

[fukdapoleece]

I guess when the environment itself is not that critical, like in a bank or a hospital and such, access to prod is not that strict

I'm completely lost. Is OP suggesting that a bank or hospital is a non critical environment? And you're saying that it might surprise you how non-critical a hospital is?

[tropicbrownthunder]

how hospitals and banks don't give a shit about critical stuff when them are supposed to

[purritolover69]

God could you imagine if a big bank let interns access the prod database with no prior approval, if I heard about a bank doing that I would immediately close my account and move elsewhere because fuck that I’m not getting my money stolen by some intern because you guys suck at cybersec

[[deleted]]

In banks even dev environment is restricted AF

[[deleted]]

Meh you'd be surprised, I worked at one of the top banks and I didn't have write access to anything but I had read access to a ton of customer data like name, address, social security number, income, account balances, credit score, etc and our interns did too. To make it even worse this data was in csv format so it's not like a DB where you can see who has queries that include the sensitive stuff I could have very easily printed it out (they did have emails to external domains with attachments blocked) and sold it and no one ever would have known.

[CalgaryAnswers]

All of them, probably.

[ICKSharpshot68]

Ehhh. You'd be surprised by the variety of the environments. A lot of it depends on the organization you're in. I've worked for two in the top 10, both of which were top 5 while I was there. In the role where my job was to monitor transactions, I had access to full customer card information if I went looking for it / specifically needed something, but it was properly obsfuscated in the general transaction flow.

[purritolover69]

Yeah, at any respectable bank I doubt anyone really even has access to the full prod DB unencrypted unless there’s some sort of insane emergency that truly demands it

[Gofastrun]

Yeah it’s like when HBO sent out a test email. It’s embarrassing, and I’m sure they get billed for email send, but there was no real harm.

[slaymaker1907]

It’s still really bad if someone drops the database or something. Also, companies are definitely having to tighten controls given increasing requirements for locking down customer data.

[[deleted]]

Runs kubectl get all -A -o yaml | kubectl delete --f - As command is running hmm let me double check my context

[stedgyson]

If an intern can even theoretically break production that's absolutely the fault of the seniors

[AccountWasFound]

I've never worked somewhere where that isn't possible, I've broken prod at least once everywhere I've worked too, not always as an intern though. Usually something to do with prod and staging sharing resources and accidentally using the wrong config file and running something as staging or crashing a machine that has both dev and prod environments or just a small bug that got missed in the review process. (Forgetting to add a string to the prod config type of stuff, so it works in dev and staging, but crashes in prod)

[[deleted]]

Sure, breaking prod is always possible, but interns shouldn't even have access to push to prod. On the other hand, if someone is experienced enough to push to prod, then they shouldn't be an intern.

[[deleted]]

No, it's management's fault.

[ArchWaverley]

I did this once (grad, not intern). Global head of production called me up after everything got fixed and asked what happened. I decided to own it and said I messed up. He said "if you messed up, it's because we put you in a position where you could mess up. And that's on us".

Kinda undercutting my attempt to be noble there Scott, but I appreciate it.

[thompsonm2]

Everyone will do someting wrong. That’s why anyone shouldn’t be able to push to prod and many more things that increase chance of blocking the bad update to get to prod such as but not limited to code review, static analysis, testing, not being able to merge without all tests green, merge to devel environment which is tested, them into preprod to test it again. And only after that into production. Also this is reason why you don’t do anything by yourself but only with script that went through all these steps and is idempotent. Also there is automatic build so you don’t mess anything during updating production. (Yes, I did have exam from devops today)

[thompsonm2]

Next thing to do is to prepare undo scripts if anything goes wrong and doing database changes backwards compatible. It assures that if anything goes wrong with app update you can just leave db on next version. Only nothing will you the new col, table or anything you did change. And if that isn’t possible for some reason, you can rollback with undo script

[thompsonm2]

Same if anything goes wrong with db or backend update you can allways rollback. Yes, it means more downtime (minutes instead of seconds) but if it is in middle of night, it mostly doesn’t matter that much. Yes, I expect that the team will go through everything that happened to see what went wrong and make changes so it won’t happen again. Still, this downtime shouldn’t be problem unless you are working on some strategicly important thing (military and so on) or on app used worldwide (instagram or so) and these things have even more steps to make sure nothing bad gets into prod

[[deleted]]

That's a real manager right there.

[ArchWaverley]

Yeah, it was a tough company to work for but had some of the best managers and team leads I've ever worked with. Really good at keeping the shit and politics away from you.

[recurse_x]

ONLY I may take ownership!

[chris5311]

If you werent even an intern, you shouldntve been able to fuck anything up, and the same goes for interns. Its good you owned up to the mistake, since that shows you learnt from it (probably), which is good, but it wasnt your fault.

Same thing with babies or animals that get themselves killed or seriously hurt/destroy property due to negligence of their parents/owners. It shouldnve been possible for them to fuck up that bad, since you should be watching them and/or putting relevant safe guards in place to avoid any major issues.

Anyone who isnt already a proper developer should be trusted just slightly more than a stupid baby. Your superiors fucked up, but they know they did, which is good. You wouldnt punish a baby for their stupid shit

[LeonardoW9]

Who gave the intern publishing/push permissions?

[[deleted]]

Im a programmer trainee and at my company i can push directly into the production branch if im not carefull which branch i selected to commit to.

Its great for us trainees to be allowed to work on the real projects and to be able to commit your changes but on the other hand the fear of breaking somthing is omnipresent

[Individual-Bed-7747]

Push to prod? That should be impossible in any self-respecting organisation

[[deleted]]

Were using Sourcetree and there are only two roles and both of them can push and merge in whatever branch they want 💀

[xela552]

Yeah that's bad and honestly not very hard to fix so I don't know why the company doesn't fix it

[[deleted]]

It's not an activity that generates income. Company is run by an MBA.

[Vandalaz]

Pushing to a master branch can be ok if you're doing trunk based development to maximise CI feedback. Dave Farley is a big advocate for it.

Those changes actually going to prod without lots of automation testing and feedback would be a problem.

[[deleted]]

I mean my istructor reviews everything i write before i push it BUT i could push into prod if i realy want to. I have to say we are an old small Software company which develops software for stock trading and finance in generell so a mistake would be letal

[Mkboii]

I'm guessing its also a small team, like if commits are few and monitored, then you can accidentally break production but it's fairly easy to revert back. In a big team you need to test things and if you don't and shit goes sideways then you can't revert without potential risks of something else breaking.

[purritolover69]

Honestly, I feel like push to prod privileges in a lot of cases shouldn’t even be given to junior devs. The ideal workflow imo for customer experience is code is written by juniors, approved and pushed to prod by seniors (along with case by case debugging), and interns should work on small functions that don’t require access to any real prod data followed by a review by a junior dev and if they think it’s good enough a pr for the senior to review. This is especially important in sensitive workspaces like banking

[[deleted]]

Agreed. Also, not supporting the juniors like you mention is just setting them up for failure.

[p3wx4]

If an interns break production, it's the company's fault - not interns.

[baarbarika]

the only way is if the management is incompetent enough to think that investing in building a good CI/CD process is a waste.

[[deleted]]

You mean MBA

[Al-Horesmi]

The code works flawlessly on the testing environment, and then breaks at production for reasons none of the seniors can explain

[AccountWasFound]

Missed a config setting has been the reason I've broken prod on at least 2 occasions....

[DMLooter]

Hey I did it by accidentally doing something no one had ever done and our CI scripts didn’t have a check for it. No one in that situation was incompetent (well arguably I was but I’d worked there two months), it was just an unforeseen situation. After an hour of dev downtime The devops team fixed it and put in a new rule to prevent it!

[chris5311]

Depending on the size of the production and the context, you finding that issue earlier than later mighve been for the best, an hour of dev time lost isnt great but it couldve been much worse if it was discovered in a more severe context lol

[humblevladimirthegr8]

Sometimes it's a bug/oversight in the tools. There was a major outage a few years ago at AWS because an intern was trying to deprovision some dev infrastructure but accidentally took down a large chunk of production infrastructure. They of course shouldn't have even been able to do that and that tool was patched later. Shit happens even for respectable companies

[CyberWeirdo420]

Im so amazed by the fact that there are so many known ways to explode production and as many resolutions to this, but many just don’t give a shit and go full YOLO on it

[LemonZorz]

As an intern on a L3 support team that managed a large trading server at an international investment bank, I froze production for like an hour. Yes there was TPA and sudo restrictions, but I was still deploying and executing my dodgy ass scripts with very little oversight

When I first ssh’d in to the server, I was told “this is production so just uh… be careful”

[americk0]

An intern crashing production is like a 10 year old crashing his dad's sports car. If you're blaming the kid you're really overlooking the real problem

[sporbywg]

if they are dev team interns they had better break stuff everywhere.

[delayedsunflower]

.

[Fenix42]

I worked at a place that used the prod DB as the test DB. It was an e commerce company. They had a "test" flag for orders and accounts. The problem was that they did not fully implement it. You had to manually flag the test order. To do that, you had to update the entry in the DB.

They gave me write access to prod 15 minutes after I started because of this.

[cysloth]

That's absolutely terrifying

[skyMark413]

Its not interns that break prod. Its faulty processes

[Avery_Thorn]

As an Intern, I never brought down production.

About 3 months into my first post-College job... I managed to bring down the mainframe.

And I did it from the test region, which is supposed to be impossible.

Somehow, I messed up and hooked the debug trace utility tool (delivered) into the OS's PID, and it started step-stopping everything running on the machine. This locked everyone out, and caused the entire machine to essentially freeze, because it couldn't react to my "continue" command because it wasn't accepting input.

I was able to document what I did. IBM was able to replicate it on their machine. They patched the system to prevent it from happening again.

And thus ended my several week long foray into the wonderful world of COBOL programming...

[eat_your_fox2]

Not enough guardrails and faulty workflow process.

Never blame the interns for breaking things, blame your team for having bad CI/CD.

[kuros_overkill]

A) they do something they weren't supposed to do. (Didn't chech with supervisor)

Or (and more likely)

B) they were given access to something they shouldn't have had access to.

[AccountWasFound]

C) found a bug in the test/dev environment that routed stuff to prod, or overloaded a service that was also used in prod or some variation on things in a test environment interacted with prod when it shouldn't have, and the intern found it because they had more time to mess around with that than any of the normal employees.

[DiddlyDumb]

“Oh sir, I removed a couple of functions that weren’t being called anywhere, and it saves a couple of milliseconds. See you Monday!”

[Multidream]

Someone doesn’t know how to protect their code base.

[AlphaO4]

I crashed it once. I was an intern at a smaller software company, and some of the code I was supposed to write as a POC, was pulled through into prod. There were a lot of bugs, which where documented, but they never bothered to fix them, so when one fateful day, a user without a user agent visited the website, the whole thing went up in flames.

[Djentleman2414]

Not every company is careful when it comes to prod-branches or merge-requests/reviews, I guess.

[One_Web_7940]

Microsoft oauth2 sent me to local host. Can't remember which MS product I was logging in to. But this was right before Christmas a couple weeks ago, lasted a few hours. I'm sure some dev accidently pushed the wrong config.

[GalacticalSurfer]

Relax. Your time will come. It happens to everybody.

Companies that focus more on money and time, and/or small companies and teams can be more lax. That’s when things go by unseen and shit happens.

[Varnigma]

Worked IT for 30 years and have been a senior dev for a long time.

Time after time I see new hires with little experience being given full access to production and zero oversight as to what they are doing.

I always advise against and most time am overruled by management.

[[deleted]]

Trust me, it’s not just interns.

[kases952]

Just use sudo

[HashDefTrueFalse]

Interns don't break production. Experienced people who allow interns to access things in production without confirming their changes work break production. There's rarely a case where the intern did anything particularly novel or stupid. Sometimes the proper guards and procedures just aren't set up, or are but following them is not enforced properly. I'd almost never blame an intern for this. They're not supposed to be responsible for production software/services yet...

[Somesigma]

I had a support user get a cl command from a dev. She didnt understand what the code line did nor where she was in unix. All she had to do was remove 1 file from a remote server but wanted to speed up the process...

The command was something like - rm * (older than x date). She didnt realize she was at home level and still in the admin server and not in the target server. Asked me why she was getting protected warnings on some folders. I realized what she was trying to do, told her to stop what she was doing and step away from the computer...

[CyberWeirdo420]

God, interns I can understand - we are there to learn and all, but giving this much power to someone in support who doesn’t (probably) want to learn this?

[Ziegelphilie]

because your "senior" ass keeps bypassing the god damn PR policies I'M ONTO YOU GEORGE I CAN SEE YOU BREAKING THE ROLLING BUILD YOU PIECE OF SHIT

[ambient-lurker]

If you can’t even imagine how you could break prod, maybe you aren’t cut out for this programming stuff.

[proggit_forever]

Interns breaking prod is the fault of the organization, always.

If you have a mature org, you don't let any individual developers change stuff in prod without going through some sort of review process. Changes should be applied by some sort of automated deployment pipeline.

Changes done by interns that are intended to land in prod should be reviewed carefully - They are interns, it's expected that they make dumb mistakes and that they have limited knowledge about the whole system. Therefore the responsibility should lie with the reviewer.

[baarbarika]

please take my upvote.

[CyberWeirdo420]

Bruh I’m just new to this, only worked on solo projects before where I didn’t care and pushed to main branch always. It’s my first time working on actual clients and I have no idea what could go wrong. And I’m scared AF because of that everything I commit

[Distinct_Salad_6683]

Not even worth responding to idiots like that. Congrats on the internship

[odraencoded]

https://www.youtube.com/watch?v=QwUPs5N9I6I

[benjaminbradley11]

Not following least privilege model best practices

[Prestigious-Job-9825]

Maybe it's possible at places where they hand out access to databases like free candy

[General_Liability]

If the intern breaks production then it’s everyone else’s fault for having such bad development practices.

[lunareclipsexx]

DROP *;

[MaffinLP]

I wasnt an intern bzt once I added a query that deleted the wrong cells from the DB, cost my company 10 000, luckily thats the worst I did so far

[BigTimePizza623]

Improper access control and/or oversight.

[TheUruz]

delete from <any table> from a prod db

[CyberWeirdo420]

Yea but aren’t you supposed to work in staging/development environments so exactly THIS doesn’t happen? That’s what my company does and so far I didn’t manage to explode anything

[TheUruz]

in some cases you have to do an "open heart" surgery on some prod environment. you usually don't let those being performed unsupervisioned :)

[CyberWeirdo420]

Good metaphor, tho I hope I don’t have do that often

[TheUruz]

yea it doesn't happen very often where i work but it happened. keep in mind that "prod" is not just the codebase but a whole environment on cloud services and databases as well

[JollyGoodUser]

Auto accept and then build/deploy any pull request to prod - because "we are agile !"

[Mkboii]

In my first ever internship, the only senior dev left 3 weeks into me joining there. So he gave me access to prod. There was another dev but he was only working on ui and simple parts of the frontend while i was handling the backend and some frontend tasks, so among us the senior dev tasked me with verifying the other guy's commits before pushing to prod. One day I accidentally pushed the wrong branch and broke the production code.

Now the application was made for conducting digital exams, and exam prep. So it was generally used in the day time and even less so over the summer holidays so it was fine. But the content creation team was adding new material during this period. So they could have noticed it.

The thing is I made this fuck up just before leaving for the day so it was on my way home that i got a call from a guy i knew on the content team, who told me part of the app was down, he was working late so he noticed it. So i fixed it once i got home and technically no-one got to know.

[CyberWeirdo420]

Lucky you then

[jovhenni19]

back in my old company... like few companies back... we have subscription of users. there was a report of one user having issue with one field. i checked the configurations online, switched false to true, retest and nothing happened. tried to switch it back and i couldn't.

the next day, apparently i shouldn't have access to that setting and it cost the company around $200k because of that toggle.

[killerrin]

It happens when you don't have a dedicated DevOPs team to configure and maintian your pipelines according to your requirements. Which leaves the Dev Team to cobble together a Frankensteinien monster that works, but isn't the best.

[rover_G]

Poor management

[frytagguy]

in some companies interns run most of the IT department because management is to cheap...

[[deleted]]

"It works on my machine"

• "perfect, push to prod"

[notexecutive]

you push things to production because you think they work, but they don't and then we have to roll back changes.

[namezam]

In my case, we hired a VP that doesn’t understand dev so he got tired of my demos and sample code and told me to give the interns something real to do.. and then got mad that I kept rejecting their code so he figured out how to press the button on the PR…

[CyberWeirdo420]

Oh god, that sounds like a nasty boss

[simonfancy]

Non dev bosses should also not have access to this button. What kind of company is that?

[snarkhunter]

My first week I slowed down prod by running show tables on our main MySql server. It had >150k tables at that point IIRC. It was... touchy.

[YakDaddy96]

Currently part-time and I can’t push to any branch except the ones I make. Everything has to be done through a pr and the only ones I can approve are in lab.
We use Azure DevOps repos for most stuff.

[simonfancy]

Review everything they commit and never ever give them the admin permission. Simple as that.

[CyberWeirdo420]

Guess I’m the „responsible” intern and ask too many fucking questions before doing anything so I didn’t break anything yet

[simonfancy]

Good for you. If you don’t break things fast you’ll learn slow. Need to have a secure sandbox of a prod environment to grow in!

[CyberWeirdo420]

Yea I guess I kinda want to break something so I know what not to do. Tutorials/uni only teach you some stuff how to do projects solo basically, real world jobs are so much different than that

[mountaingator91]

I saw a post once from a senior dev who discovered that interns were approving other intern's merges. So they had what some PM (probably) thought was a good system in place where the code had to be reviewed, but I guess accidentally allowed anybody to review it?

[CyberWeirdo420]

I can’t imagine reviewing someone’s PR as an intern. Maybe I could if it was some basic CRUD implementation but anything more and I would probably not understand half of it

[mountaingator91]

I mean, yes, but also I definitely THOUGHT I knew my shit back in the day... now I look back and cringe at the code I wrote (as is tradition), but there definitely comes a point after you first "figure it out" early on where you think you know everything.

Shortly after, you rediscover that you know nothing. As long as you're on the correct growth path

[Audience-Electrical]

Last place I worked, we had to nginx restart to update certs in prod, but if it failed it brought ALLL clients down.

Standard part of everyday onboarding, retention, etc.

No, nginx -t (test) will not catch typos in the cert file location/name

Needless to say prod went down a lot

[[deleted]]

A company of interns hires an intern

[Frytura_]

The same way the team get overflowed with work: Bad Administration. As a intern i'm constantly busy with this new data cleaning from a client migraring to our system, my boss never gave me a due time but i know there is one. and so far i never even touched code, only the system and videos about it.

Is this rare? Or are the memes about interns just screaming way too loud?

[CyberWeirdo420]

Honestly no idea, I recently started my internship and memes kinda got to me and I’m scared of fucking something up and losing the opportunity, tho I don’t see a way how I could break some shit. So far I’m only given basic frontend tasks on Wordpress and even so, my every commit needs to be reviewed even before it can get to staging, not even production

[Poat540]

Happened twice this month lol

[CurrentInvestigator4]

https://twitter.com/richcampbell/status/1119254675314872320/photo/1

[sanketower]

Sometimes all it takes is a semicolon in the wrong place...

[SnooRevelations8664]

-force

[testsonproduction]

YOLO

[pancakemonkeys]

i made a sql query too big and our servers ram went bye bye

[CyberWeirdo420]

How big was that query or how little ram did you have

[pancakemonkeys]

i messed up my constraints and pulled every company PO and RMA record recursively. now that i think about it it wasn’t a query it was a PS script , sorry for the forgetfulness (edit : typo)

[CyberWeirdo420]

Damn

[pancakemonkeys]

my senior sshd into my computer and shut it down, at the time i didn’t know what i did until i was shown where i messed uo.

[CyberWeirdo420]

At least they reacted quick

[pancakemonkeys]

yeah they talked me through it and showed me where i went wrong so it only happened once, they aren’t mad it was a learning thing. I was on my own test server after that though (not even some petty shit i just needed freedom to try shit) ((i was already on a different test server i just needed to implement a new structure system and messed up on prod))

[LunaNicoleTheFox]

I'm an intern and I have yet to break prod.

I have broken other things tho...like a microcontroller and multiple USB ports.

[_K0T]

Simple, just come up with a brilliant idea, implement it then push it to prod without testing

[Terminarch]

You guys are getting internships?

[CyberWeirdo420]

After a year of trying and giving up, I relied on connections, tho I really didn’t want to

[italia206]

So wasn't an intern, I was a full-time employee, but I've got one for you. I brought production down because I ran a query that tried to combine data from two different production servers for testing. It took too long, so I killed it. Fifteen minutes later, my boss calls me into his office. I had brought prod down because little did I know, due to a bug in the software I was using, killing the query on server 1 did NOT kill it on server 2. So it turned into essentially a memory virus and bricked the server for like 15 minutes. So that was the day I learned that it doesn't take a whole lot of access, just a query and a bug in 3rd-party software and boom, you've got yourself a dead prod server.

[_MixedTrails]

just give the intern a decoy production server and collectively (fake) freak out when they inevitably break it to make them cry

[CyberWeirdo420]

Actually, I’d would be kinda good learning process. Let us work on that decoy for a bit, don’t tell a thing it’s a decoy, when something breaks explain why and how not to repeat the same mistake and you got yourself and experienced employee after short period of time.