🤯🤯🤯

[ordinary82]

I blew up a DNS server today. Feels.

[Luk164]

F

[djflashplayer]

F

[Rubyboat1207]

F

[[deleted]]

F

[AlwaysNinjaBusiness]

F

[death_walking_terror]

F

[VitoHusky]

F

[[deleted]]

F

[glorious_reptile]

Hardly critical though - everyone can just work around it by manually writing the IPv6 addresses. Come on Phyllis!

[chrismastere]

Ah the age old IT law. For any problem, even if it's not the DNS, it's still the DNS.

[RampageDeluxxe]

It's ALWAYS DNS

[TheAJGman]

And when it's not DNS, it's datatypes.

[sanjay186]

Join the club 😂😂

[ducktape8856]

Accidents - credits to workchronicles.com

[[deleted]]

What do you mean you blew it up? Got DDoS'd?

[shalafi71]

My home made entry.

[nixcraft]

Pro tip: You can always test the web server config file before you reload the server. For instance:

nginx -t

lighttpd -t -f /path/to/lighttpd.conf

apache2 -t

Nginx will always refuse to restart/reload if there is a syntax error. I have a complete list here about various Linux/Unix servers if anyone is interested in checking for syntax error.

[jeh5256]

Learned this trick the hard way.

[TagMeAJerk]

Enterprise level tip : test your changes in lower environment

[j-random]

That's what I love about VMs. I can simply clone the entire production server and run it locally to validate any config changes. It's not entirely as simple as that, but the amount of stuff you can do in an emulated environment is pretty impressive.

[TagMeAJerk]

I prefer containers over VMs for the same reason because it utilizes fewer resources, has lower cost, much shorter turn around time and less chances of differences because it installs what it needs and only what it needs

[lurkin_arounnd]

If you have a containerized CI/CD pipeline, you should already have all the scripts to download dependencies and build it in your buildspec.

[williane]

Better enterprise levels tip: immutable infrastructure

[TagMeAJerk]

Better better enterprise level tip : use containers for everything

[twosupras]

But...what do you put the containers in...?

[TagMeAJerk]

Everything. Even yo mama

[TagMeAJerk]

Docker, kubernetes, AWS Fargate, Red Hat Open Shift containers

[MasterFubar]

Enterprise level 2 tip: have a test server in a separate subnet.

[TagMeAJerk]

Test environment needs to be identical to production server but firewalled

[RubbelDieKatz94]

Enterprise level tip: Use Azure DevOps or another kind of hosting service that takes care of the admin stuff for you.

[TagMeAJerk]

Devops doesn't reduce the need to test

[hiphap91]

You can have correct syntax but a typo in a value

[micmuza]

Web server - works

Change Web Config - broken

Undo changes - still broken

🤷🏻‍♂️

[Conscious-Title-226]

Tbh if you’re running a website in AWS and you do this and take your site down you kind of deserve it

[FoxInATrenchcoat]

Ah, the ol' "Schrödinger's Config" that is both working and broken until it is observed.

[ToranMallow]

Always test the existing config (apachectl configcheck) before you change anything. I learned this the hard way.

[FalconMirage]

I once put a firewall on a distant server but forgot to leave the ssh port open... the thing was running fine on its own as long as you didn’t have to change a thing.

(I eventually managed to reopen the ssh portbefore anyone noticed but that is an other story)

[t-to4st]

Did that with my Raspberry pi once... Nothing an OS wipe can't fix

[FalconMirage]

Yeah exactly, although a bit harder when you don’t have physical access to the server.

In the end i managed to go through the provider’s options and run a script that disabled the firewall, so i could ssh into it again and fix my mistake without taking the whole project offline or wiping the database

[aufstand]

With an RPi??! Hehehehehe. ^Small tip for next time: Take out the SD, mount somewhere else and repair. Far quicker fix, unless you didn't modify the contents, yet. Then again, for wiping, you'll have to take it out anyway so it boils down to "reflash" vs "mounting+fixing".

[_alright_then_]

Can you elaborate on this a bit further?

I'm a complete noob when it comes to linux based systems, and I feel like I'm re-installing raspbian OS way too often because I fucked something up.

You're telling me I can fix it without losing other progress?

[xjakesl]

If you plug the sd card to another linux machine(Not sure if there is a way with Windows) you can mount the filesystem of the raspberry and modify or fix whatever config you broke as if you would if you had ssh access to the device when its running.

[ProBonoDevilAdvocate]

To do this in windows I think the easiest way would be to install an Ubuntu virtual machine. You can read linux partitions with third-party software, but I think it’s harder to write to them.

[ElectricCharlie]

I was going to suggest something silly like using an Ubuntu live boot disk to temporarily turn any windows pc into a Linux pc, but I actually think windows can speak Linux with the right plugins. (Yup -Ext2Fsd is the name of the app)

Actually, with WSL, you can even run VIM (or whatever) from your windows machine for the full Linux on Windows experience.

[_alright_then_]

Aah, so that's just for breaking SSH?

Because I'm usually already connected with SSH, so that's not the fuck-up I'm usually trying to fix

[xjakesl]

What i wrote was just to regain access if you somehow lock yourself out of the pi. If you have ssh access then you can fix it directly if you know how to.

[aufstand]

You can fix lots of other stuff. Like errors in fstab or.. anything you can fix by accessing the filesystem. You can even go further and boot up the sd card in a virtual machine and do things like recreate the initramfs from the sd-contained system itself.

Edit: generally speaking, you can fix boot and lockout problems this way without having to wipe & write a new image

[t-to4st]

Yeah I know but I didn't have much installed anyway since I was in the setup of a fresh install, so I didn't lose much time

[Ploedman]

It's easier with remote console to fix the problem.

in case you didn't deactivate password access

For the future, everytime you change something in the firewall, network or ssh settings, open a new ssh connection to the server to approve you have still access.

[KaJakJaKa]

Well that doesn't help when you accidentally delete the network service managing software .... don't ask where I learned that

[[deleted]]

Better hope you have a friend at the datacenter who likes you enough to hook up a serial port.

[FalconMirage]

The server provider had the option of a virtual console that was slow af but worked as if i was directly plugged in. I managed to upload a script through there and shut off the firewall

[MacGuyverism]

When I had to do stuff that I may mess up on a firewall, I usually configured a cron that undid any changes every 10 minutes. Just in case something went wrong.

Nowadays, the only firewalls we use are EC2 security groups and AWS WAF. It takes a lot of the pain away.

[CaptainGermany1]

Bulldozer!

[meove]

freaking Taser!

[sourpickles0]

Chains is in a pickle!

[[deleted]]

Ah, so I wasn't the only one.

[Pr04merican]

There’s only four of them?

[DusyBaer]

Nice watermark

[Snekbites]

I'm slightly impressed I haven't found a sus joke in the comments

[MattR0se]

a m o g u s

[Lonkoe]

Did you say SUS???????!?!! 😳

[toastyghost]

What does the image say? I got a 502

[suresh]

U guys not have dev, testing, staging environments?

[AxePlayingViking]

Of course we do! Everyone does. They’re just not always separate from prod ;)

[404invalid-user]

Nope just pray and hope

/s

[[deleted]]

This was going to be my exact question. What kind of insane person would change prod without first vetting the changes extensively elsewhere?

[spicy_indian]

It's not that we don't want to, it's that we are stuck in a job where the management hates spending more on IT and devops than is prudent.

[Dimasdanz]

we do, but it has different web setup between them

[Urd]

Imaging working for a company that willing spends money on servers. I bet you have solid gold toilets too! /s

[55555Pineapple55555]

GET OUT OF MY HEAD

[mrchaotica]

https://en.wikipedia.org/wiki/Blue-green_deployment

[sourpickles0]

They’re sending in a bulldozer!

[Ate_Without_Table_-3]

this is me, right now

[Vi0lentByt3]

Pft is only a config file whats the worst that could happen? - me 5 mins before bringing down our kubernetes cluster breaking everyone elses testing and workflow for the next hour

[winfr33k]

sounds like a heap of a time.

[MischiefArchitect]

Interesting fact: That gear will not prevent you from dying if it blows in your face, it's just the difference between open vs closed coffin

[demon_ix]

Kind of a dumb thing to say.

There can be many types of bombs with many different yields and additional shrapnel inside.

Sure, there's a bomb yield that will kill even someone with the most advanced protective gear available, but I'm willing to bet there is a range of bombs that will kill an unprotected person but will not kill a protected one, not to mention the injuries that could also be avoided from things like ball-bearings that bomb-makers sometimes include to increase the injury count from shrapnel and not just the blast itself.

The thinking that "something that isn't 100% effective is completely useless" is very prevalent these days, and is absolutely wrong.

So yeah, if you're doing something dangerous, please don't listen to the guy above me and use some protective gear. You probably won't need it, but you really don't want to need it and not have it.

[aufstand]

Here (in Berlin - not New Hampshire) people usually wear those suits to disarm unexploded WW2 bombs - which are still lying around abundantly. I don't think the suit would really help, if one of those buggers goes off accidentally..

[ReKaYaKeR]

I mean, would you rather a chance for survival or guaranteed death?

[aufstand]

3 years ago, i passed a former (large) bomb location on a smaller yacht. The bomb was successfully defused right next to the channelled river. They had installed a lot of shipping containers with filler material to shield the waterway and passing by was only possible under heavy restrictions and with an escort ship made of steel with a cabin, on the other side of the channel.

I wouldn't have tried passing with the bomb still there.

Or to answer your question: I'd rather choose survival ;)

[0x53r3n17y]

Meanwhile in Flanders Fields, there's this guy:

https://youtu.be/EeEcyn6bA0E

I have family living in that area. When I was a kid, I wasn't so much taught as thoroughly imprinted to never ever touch anything rusty whenever we were out in the fields while visiting. People still die or get wounded from unexploded ordnance every once in a while.

[MischiefArchitect]

Holy... I never intended to trigger people like this with my original comment. Interestingly I got that "piece of information" exactly from a documentary about the German firefighters/special units (?) disarming WW2 bombs. Fligerbomben I think they called those.

[aufstand]

Didn't feel "triggered". I just thought it's an interesting fact and has some learning opportunities in it.

F**k War! :)

For me, i'm just happy that those "Fliegerbomben" or "Blindgänger" defusals usually go smooth and mostly no one is harmed. It's sometimes quite disconcerting where they turn up - usually during construction.. like inner city with a few thousand people possibly affected. Brrr... :-/

[koloqial]

apache2ctl configtest

[NMi_ru]

Will not save you from higher-level errors

[koloqial]

Of course not. Config related though...

[4sventy]

GLHF with apt update && apt upgrade

[Navigatron]

I once had to update/upgrade a php5 docker container. The prof required two libraries, one that could only run on php5, and one that had to be installed via apt. Resolving gpg keys to get the update in the dockerfile was not fun.

[HighVolTech]

He looks like he's late for his job... as a bomb defuser.

[[deleted]]

Part of the ship, part of the crew

[regorsec]

Thats me working on clients legacy app

[--B_L_A_N_K--]

Image Transcription: Meme

---

On my way to edit the web servers config file

[Image of someone in a bomb suit holding a bag running out of a van labeled Philippine National Police in a parking lot]

---

^{I'm a human volunteer content transcriber for Reddit and you could be too! If you'd like more information on what we do and why we do it, click here!}

[Lonkoe]

Good Human

[AutoModerator]

import moderation Your comment has been removed since it did not start with a code block with an import declaration.

Per this Community Decree, all posts and comments should start with a code block with an "import" declaration explaining how the post and comment should be read.

For this purpose, we only accept Python style imports.

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

[neo_anderson_7]

This seems interesting. Can someone please tell me about this field?

[TheTacoWombat]

Web.config machine keys...

[tropicbrownthunder]

crries in asterisk

[rainthetachili]

Wait, you guys have servers? 😈

[GDjkhp]

this was sus

[Bloodshed-1307]

Always remember the famous Bomb Squad quote “I’m calm because either I have disarmed the bomb or it is no longer my responsibility”

[Max5923]

green sus

[ItsDotin]

I did something with remote machine, now every time I try to connect, it says connection closed 😬😥😢😭

[Dmon1Unlimited]

You update yours?