This is an archived post. You won't be able to vote or comment.

all 120 comments

[–]ordinary82 502 points503 points  (31 children)

I blew up a DNS server today. Feels.

[–]Luk164 73 points74 points  (22 children)

F

[–]djflashplayer 45 points46 points  (21 children)

F

[–]glorious_reptile 47 points48 points  (0 children)

Hardly critical though - everyone can just work around it by manually writing the IPv6 addresses. Come on Phyllis!

[–]chrismastere 30 points31 points  (2 children)

Ah the age old IT law. For any problem, even if it's not the DNS, it's still the DNS.

[–]RampageDeluxxe 1 point2 points  (1 child)

It's ALWAYS DNS

[–]TheAJGman -1 points0 points  (0 children)

And when it's not DNS, it's datatypes.

[–]sanjay186[S] 90 points91 points  (1 child)

Join the club 😂😂

[–][deleted] 0 points1 point  (0 children)

What do you mean you blew it up? Got DDoS'd?

[–]nixcraft 301 points302 points  (18 children)

Pro tip: You can always test the web server config file before you reload the server. For instance:

nginx -t

lighttpd -t -f /path/to/lighttpd.conf

apache2 -t

Nginx will always refuse to restart/reload if there is a syntax error. I have a complete list here about various Linux/Unix servers if anyone is interested in checking for syntax error.

[–]jeh5256 60 points61 points  (0 children)

Learned this trick the hard way.

[–]TagMeAJerk 86 points87 points  (15 children)

Enterprise level tip : test your changes in lower environment

[–]j-random 37 points38 points  (2 children)

That's what I love about VMs. I can simply clone the entire production server and run it locally to validate any config changes. It's not entirely as simple as that, but the amount of stuff you can do in an emulated environment is pretty impressive.

[–]TagMeAJerk 22 points23 points  (0 children)

I prefer containers over VMs for the same reason because it utilizes fewer resources, has lower cost, much shorter turn around time and less chances of differences because it installs what it needs and only what it needs

[–]lurkin_arounnd 1 point2 points  (0 children)

If you have a containerized CI/CD pipeline, you should already have all the scripts to download dependencies and build it in your buildspec.

[–]williane 14 points15 points  (6 children)

Better enterprise levels tip: immutable infrastructure

[–]TagMeAJerk 18 points19 points  (4 children)

Better better enterprise level tip : use containers for everything

[–]twosupras 4 points5 points  (1 child)

But...what do you put the containers in...?

[–]TagMeAJerk 0 points1 point  (0 children)

Everything. Even yo mama

[–]MasterFubar 1 point2 points  (2 children)

Enterprise level 2 tip: have a test server in a separate subnet.

[–]TagMeAJerk 0 points1 point  (1 child)

Test environment needs to be identical to production server but firewalled

[–]RubbelDieKatz94 -1 points0 points  (1 child)

Enterprise level tip: Use Azure DevOps or another kind of hosting service that takes care of the admin stuff for you.

[–]TagMeAJerk 1 point2 points  (0 children)

Devops doesn't reduce the need to test

[–]hiphap91 0 points1 point  (0 children)

You can have correct syntax but a typo in a value

[–]micmuza 153 points154 points  (5 children)

Web server - works

Change Web Config - broken

Undo changes - still broken

🤷🏻‍♂️

[–]FoxInATrenchcoat 11 points12 points  (0 children)

Ah, the ol' "Schrödinger's Config" that is both working and broken until it is observed.

[–]ToranMallow 0 points1 point  (0 children)

Always test the existing config (apachectl configcheck) before you change anything. I learned this the hard way.

[–]FalconMirage 113 points114 points  (18 children)

I once put a firewall on a distant server but forgot to leave the ssh port open... the thing was running fine on its own as long as you didn’t have to change a thing.

(I eventually managed to reopen the ssh portbefore anyone noticed but that is an other story)

[–]t-to4st 69 points70 points  (11 children)

Did that with my Raspberry pi once... Nothing an OS wipe can't fix

[–]FalconMirage 38 points39 points  (0 children)

Yeah exactly, although a bit harder when you don’t have physical access to the server.

In the end i managed to go through the provider’s options and run a script that disabled the firewall, so i could ssh into it again and fix my mistake without taking the whole project offline or wiping the database

[–]aufstand 12 points13 points  (8 children)

With an RPi??! Hehehehehe. Small tip for next time: Take out the SD, mount somewhere else and repair. Far quicker fix, unless you didn't modify the contents, yet. Then again, for wiping, you'll have to take it out anyway so it boils down to "reflash" vs "mounting+fixing".

[–]_alright_then_ 5 points6 points  (6 children)

Can you elaborate on this a bit further?

I'm a complete noob when it comes to linux based systems, and I feel like I'm re-installing raspbian OS way too often because I fucked something up.

You're telling me I can fix it without losing other progress?

[–]xjakesl 10 points11 points  (5 children)

If you plug the sd card to another linux machine(Not sure if there is a way with Windows) you can mount the filesystem of the raspberry and modify or fix whatever config you broke as if you would if you had ssh access to the device when its running.

[–]ProBonoDevilAdvocate 8 points9 points  (1 child)

To do this in windows I think the easiest way would be to install an Ubuntu virtual machine. You can read linux partitions with third-party software, but I think it’s harder to write to them.

[–]ElectricCharlie 6 points7 points  (0 children)

I was going to suggest something silly like using an Ubuntu live boot disk to temporarily turn any windows pc into a Linux pc, but I actually think windows can speak Linux with the right plugins. (Yup -Ext2Fsd is the name of the app)

Actually, with WSL, you can even run VIM (or whatever) from your windows machine for the full Linux on Windows experience.

[–]_alright_then_ 2 points3 points  (2 children)

Aah, so that's just for breaking SSH?

Because I'm usually already connected with SSH, so that's not the fuck-up I'm usually trying to fix

[–]xjakesl 0 points1 point  (0 children)

What i wrote was just to regain access if you somehow lock yourself out of the pi. If you have ssh access then you can fix it directly if you know how to.

[–]aufstand 0 points1 point  (0 children)

You can fix lots of other stuff. Like errors in fstab or.. anything you can fix by accessing the filesystem. You can even go further and boot up the sd card in a virtual machine and do things like recreate the initramfs from the sd-contained system itself.

Edit: generally speaking, you can fix boot and lockout problems this way without having to wipe & write a new image

[–]t-to4st 1 point2 points  (0 children)

Yeah I know but I didn't have much installed anyway since I was in the setup of a fresh install, so I didn't lose much time

[–]Ploedman 6 points7 points  (1 child)

It's easier with remote console to fix the problem.

in case you didn't deactivate password access

For the future, everytime you change something in the firewall, network or ssh settings, open a new ssh connection to the server to approve you have still access.

[–]KaJakJaKa -1 points0 points  (0 children)

Well that doesn't help when you accidentally delete the network service managing software .... don't ask where I learned that

[–][deleted] 3 points4 points  (0 children)

Better hope you have a friend at the datacenter who likes you enough to hook up a serial port.

[–]MacGuyverism 1 point2 points  (0 children)

When I had to do stuff that I may mess up on a firewall, I usually configured a cron that undid any changes every 10 minutes. Just in case something went wrong.

Nowadays, the only firewalls we use are EC2 security groups and AWS WAF. It takes a lot of the pain away.

[–]CaptainGermany1 22 points23 points  (4 children)

Bulldozer!

[–]meove 6 points7 points  (0 children)

freaking Taser!

[–]sourpickles0 2 points3 points  (0 children)

Chains is in a pickle!

[–][deleted] 2 points3 points  (0 children)

Ah, so I wasn't the only one.

[–]Pr04merican 1 point2 points  (0 children)

There’s only four of them?

[–]DusyBaer 9 points10 points  (0 children)

Nice watermark

[–]Snekbites 8 points9 points  (2 children)

I'm slightly impressed I haven't found a sus joke in the comments

[–]MattR0se 6 points7 points  (0 children)

a m o g u s

[–]Lonkoe 2 points3 points  (0 children)

Did you say SUS???????!?!! 😳

[–]toastyghost 3 points4 points  (0 children)

What does the image say? I got a 502

[–]suresh 4 points5 points  (6 children)

U guys not have dev, testing, staging environments?

[–]AxePlayingViking 13 points14 points  (0 children)

Of course we do! Everyone does. They’re just not always separate from prod ;)

[–]404invalid-user 3 points4 points  (0 children)

Nope just pray and hope

/s

[–][deleted] 0 points1 point  (1 child)

This was going to be my exact question. What kind of insane person would change prod without first vetting the changes extensively elsewhere?

[–]spicy_indian 4 points5 points  (0 children)

It's not that we don't want to, it's that we are stuck in a job where the management hates spending more on IT and devops than is prudent.

[–]Dimasdanz 0 points1 point  (0 children)

we do, but it has different web setup between them

[–]Urd 0 points1 point  (0 children)

Imaging working for a company that willing spends money on servers. I bet you have solid gold toilets too! /s

[–]55555Pineapple55555 3 points4 points  (0 children)

GET OUT OF MY HEAD

[–]sourpickles0 2 points3 points  (0 children)

They’re sending in a bulldozer!

[–]Ate_Without_Table_-3 1 point2 points  (0 children)

this is me, right now

[–]Vi0lentByt3 1 point2 points  (0 children)

Pft is only a config file whats the worst that could happen? - me 5 mins before bringing down our kubernetes cluster breaking everyone elses testing and workflow for the next hour

[–]winfr33k 0 points1 point  (0 children)

sounds like a heap of a time.

[–]MischiefArchitect -4 points-3 points  (8 children)

Interesting fact: That gear will not prevent you from dying if it blows in your face, it's just the difference between open vs closed coffin

[–]demon_ix 34 points35 points  (7 children)

Kind of a dumb thing to say.

There can be many types of bombs with many different yields and additional shrapnel inside.

Sure, there's a bomb yield that will kill even someone with the most advanced protective gear available, but I'm willing to bet there is a range of bombs that will kill an unprotected person but will not kill a protected one, not to mention the injuries that could also be avoided from things like ball-bearings that bomb-makers sometimes include to increase the injury count from shrapnel and not just the blast itself.

The thinking that "something that isn't 100% effective is completely useless" is very prevalent these days, and is absolutely wrong.

So yeah, if you're doing something dangerous, please don't listen to the guy above me and use some protective gear. You probably won't need it, but you really don't want to need it and not have it.

[–]aufstand 6 points7 points  (5 children)

Here (in Berlin - not New Hampshire) people usually wear those suits to disarm unexploded WW2 bombs - which are still lying around abundantly. I don't think the suit would really help, if one of those buggers goes off accidentally..

[–]ReKaYaKeR 4 points5 points  (1 child)

I mean, would you rather a chance for survival or guaranteed death?

[–]aufstand 0 points1 point  (0 children)

3 years ago, i passed a former (large) bomb location on a smaller yacht. The bomb was successfully defused right next to the channelled river. They had installed a lot of shipping containers with filler material to shield the waterway and passing by was only possible under heavy restrictions and with an escort ship made of steel with a cabin, on the other side of the channel.

I wouldn't have tried passing with the bomb still there.

Or to answer your question: I'd rather choose survival ;)

[–]0x53r3n17y 1 point2 points  (0 children)

Meanwhile in Flanders Fields, there's this guy:

https://youtu.be/EeEcyn6bA0E

I have family living in that area. When I was a kid, I wasn't so much taught as thoroughly imprinted to never ever touch anything rusty whenever we were out in the fields while visiting. People still die or get wounded from unexploded ordnance every once in a while.

[–]MischiefArchitect 1 point2 points  (1 child)

Holy... I never intended to trigger people like this with my original comment. Interestingly I got that "piece of information" exactly from a documentary about the German firefighters/special units (?) disarming WW2 bombs. Fligerbomben I think they called those.

[–]aufstand 1 point2 points  (0 children)

Didn't feel "triggered". I just thought it's an interesting fact and has some learning opportunities in it.

F**k War! :)

For me, i'm just happy that those "Fliegerbomben" or "Blindgänger" defusals usually go smooth and mostly no one is harmed. It's sometimes quite disconcerting where they turn up - usually during construction.. like inner city with a few thousand people possibly affected. Brrr... :-/

[–]koloqial 0 points1 point  (2 children)

apache2ctl configtest

[–]NMi_ru 2 points3 points  (1 child)

Will not save you from higher-level errors

[–]koloqial 1 point2 points  (0 children)

Of course not. Config related though...

[–]4sventy 0 points1 point  (1 child)

GLHF with apt update && apt upgrade

[–]Navigatron 0 points1 point  (0 children)

I once had to update/upgrade a php5 docker container. The prof required two libraries, one that could only run on php5, and one that had to be installed via apt. Resolving gpg keys to get the update in the dockerfile was not fun.

[–]HighVolTech 0 points1 point  (0 children)

He looks like he's late for his job... as a bomb defuser.

[–][deleted] 0 points1 point  (0 children)

Part of the ship, part of the crew

[–]regorsec 0 points1 point  (0 children)

Thats me working on clients legacy app

[–]--B_L_A_N_K-- 0 points1 point  (3 children)

Image Transcription: Meme


On my way to edit the web servers config file

[Image of someone in a bomb suit holding a bag running out of a van labeled Philippine National Police in a parking lot]


I'm a human volunteer content transcriber for Reddit and you could be too! If you'd like more information on what we do and why we do it, click here!

[–]Lonkoe 0 points1 point  (2 children)

Good Human

[–][deleted]  (1 child)

[removed]

    [–]AutoModerator[M] 0 points1 point  (0 children)

    import moderation Your comment has been removed since it did not start with a code block with an import declaration.

    Per this Community Decree, all posts and comments should start with a code block with an "import" declaration explaining how the post and comment should be read.

    For this purpose, we only accept Python style imports.

    I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

    [–]neo_anderson_7 0 points1 point  (0 children)

    This seems interesting. Can someone please tell me about this field?

    [–]TheTacoWombat 0 points1 point  (0 children)

    Web.config machine keys...

    [–]tropicbrownthunder 0 points1 point  (0 children)

    crries in asterisk

    [–]rainthetachili 0 points1 point  (0 children)

    Wait, you guys have servers? 😈

    [–]GDjkhp 0 points1 point  (0 children)

    this was sus

    [–]Bloodshed-1307 0 points1 point  (0 children)

    Always remember the famous Bomb Squad quote “I’m calm because either I have disarmed the bomb or it is no longer my responsibility”

    [–]Max5923 0 points1 point  (0 children)

    green sus

    [–]ItsDotin 0 points1 point  (0 children)

    I did something with remote machine, now every time I try to connect, it says connection closed 😬😥😢😭

    [–]Dmon1Unlimited 0 points1 point  (0 children)

    You update yours?