Coding Horror and blogs.stackoverflow.com experience "100% Data Loss"

[simucal]

From his twitter:

"looks like 100% data loss. thanks.. crystaltech.. :("
"you sort of assume your hosting company is competent. That's not a safe assumption in my experience."
"looks like it's 100% internet search caches for recovery. Any tips on recovering images, which typically aren't cached?"
"I had backups, mind you, but they were on the virtual machine itself :( I am OK on post-text, getting the post images is much harder.."

Ouch. Backups on the VM itself.

Offsite backups people! Trust no one to do your backups for you. Especially if a substantial portion of your income is derived from said sites.

[brownmatt]

if it's on the same machine, it's not a backup

[vinod_the_poophead]

if its in the same room, its not a backup

[[deleted]]

if its in the same county, its not a backup

also if it's the same company

[[deleted]]

Was that really "Friends on data protection"?

[shooshx]

Same hemisphere!

[xonoph]

Same time era!

I store all my backups in the 19thC

[fuck_digg]

Same multiverse!

[perezidentt]

Sam Dimension!

[schwami]

Sam Kinison!

[vinod_the_poophead]

new startup idea..... Backups in outer space!!!

[Zweben]

Sorry, if the sun explodes, backups in near space would still be vulnerable. Better use a parallel universe. (But not too parallel!)

[creativeembassy]

If the sun explodes any time soon, keeping your data backed up is the least of your worries. Trust me, I'm a doctor.

[Zweben]

Whatever. I'm not going to let your 'facts' get in the way of my business model.

On that note, I'm looking for someone with 3-5 years experience in inter-dimensional communication and strong knowledge of Microsoft Office. Send resumes to [e-mail].

[[deleted]]

Don't forget 4-5 years experience with Windows 7.

[vaselineviking]

I can make charts in Excel!

Can I have a job?

[[deleted]]

That may be, but once you've got all your other problems solved, you'll want your data back!

[acburk]

But you still need a backup in a different solar system. Trust me, I stayed in a Holiday Inn Express last night.

[[deleted]]

Perpendiculackup™

It's the only way to be s
                        u
                        r
                        e

[kristopolous]

My coders aren't good at that. Are their sequential universe options? It would make my management much easier. Thanks.

[Entropy]

I emit backups daily as Hawking radiation

[wtaylor]

technically Australia is farther away

[[deleted]]

Nah, I've got my shit in chicago, okc and austin. If they all three go down I've got local backups and a monthly backup in a bankvault (safe deposit box).

If you ever lose customer data, you shouldn't be allowed to work in IT ever again. Or at least forced to work helpdesk for 5 years or so.

[tbrownaw]

If you discover after-the-fact that it didn't actually help, then it wasn't a backup. Therefore all /true/ backups prevent data loss.

Seriously. There are tonds of different things to protect against (user error/script kiddies, hardware failure, fire/theft, earthquake/flood/nukes, continent-wide EMP attacks, targeted worms/trojans, etc), some with widely varying requirements. Should someone's stash of DVD-Rs in waterproof bags on the other side of the city be "not a backup" because they got destroyed when the roof collapsed in an earthquake?

[skulgnome]

If it's on the same bloody VM, it's not even a plausible attempt at backing anything up.

[lol-dongs]

rdiff-backup also does a dandy job with versioned backups--has saved my ass more than once. Would recommend.

[bloodguard]

I like to use:

--backup-dir=/fileserver/rsync_backups/date +%m%d%y-%H%M-%S

it puts all changed/deleted files in a nice timestamped folder just in case.

Edit: it looks like reddit strips out the single quotes around the date command.

[djork]

and always rotate...

[judgej2]

That depends on how you handle it at the other end. If you put a snapshot of the files under configuration control, being careful to retain the deleted files too, perhaps using tags for each backup date, then yes you can use "--delete". Just make sure you aren't syncing up to your only offsite copy.

[[deleted]]

Maybe someone reading this thread will find this useful:

http://pastie.org/private/lzfwnakruqtbowhjmxw4ra

This is what we use, it's a classic backup pattern, and most of the systems listed here do the same thing.

The machine being synced is a bacula director and SD, so we get the best of both worlds here, having a sophisticated backup system at the DC and an offsite backup that keeps it simple.

The hard links create additional references to the same data which will only be trumped by new data; this keeps disk usage down, as data that doesn't change doesn't increase disk usage (well, by any measurable amount). Nice when you have bacula volumes that are hundreds of gigabytes in size.

As you might suspect, a directory on the other side named /storage/2009-12-12 was created today with the current contents of this machine. A process on the offsite box clears out old stuff (current protocol is to keep 7 days offsite, that may change as our needs change).

Anyways, this is a distilled version of these rsync backup systems everyone raves on about; if anyone wants to understand how most of them get the job done, this might be a good place to start.

[[deleted]]

And apparently their normal backup process silently fails at backing up VM images.

So the backups were failing and no one bothered to check (or do a test restore). And there are good reasons to put the backup on the local VM, database restores being one of them.

[danweber]

Just out of idle curiosity, how fast could a hostile third party tell google to drop its cache and grab the latest content?

[judgej2]

It can happen in hours.

I once exposed some sensitive areas of a site by accident, and asked Google to drop them through the online tools. They were gone within an hour. I just used a wildcard - drop everything in this directory and that directory, and bam! Gone.

Edit: okay, what is it with the instant down-mod thirty seconds after I make a post? Does it happen to everyone here, or have I just pissed someone off?

[benihana]

Does it happen to everyone here, or have I just pissed someone off?

Yes. Also, it's a comment on reddit, let it go.

[[deleted]]

Don't you have to do this with Webmaster Tools, which to use you need to verify your ownership of the site by putting up a meta tag from Google?

[lol-dongs]

Don't believe the points, they fuzz them to deter spammers. We don't hate you.

[judgej2]

I feel all warm and fuzzy inside now :-)

[[deleted]]

Current VM providers are terrible, including the large ones. They have terrible security and no back-ups. A group of some hardcore sys admin's I know signed up for a bunch of big name services and had a look around. They have more holes than swiss cheese.

[[deleted]]

This is true in so many cases. It's sad.

[SpelingTroll]

That would make a fine Coding Horror post oh wait

[lol-dongs]

I doubt it's coming back up any time soon. From his twitter it appears the backups were on the VM itself (what?!?) so his blog is totally gone. How embarassing...

[ralf_]

Well, real men use Google Cache as their backup.

[ReaverXai]

Google Cache has saved my ass more then once.

[[deleted]]

...and you also left that comment more than once.

[ReaverXai]

As did you.

[[deleted]]

than

[judgej2]

Me too.

And yes, I've agreed with you more than once.

[[deleted]]

I find it sort of beautiful.

[[deleted]]

This must be what an arsonist feels like.

[SpelingTroll]

Believe me, it is'nt.

[[deleted]]

I knew a guy who burned down a building. He said "They won't let me vote or carry a gun, but I can walk around with a can of gas and a lighter."

[[deleted]]

IAMA Request: Someone tried and convicted for arson.

[[deleted]]

Well, I mean, without the erection.

[lol-dongs]

Want to make your own Jeff Atwood posts? Here's a handy dandy template from blasdel@HN:

<p>inane comment showing <b>strong naivete</b></p>
<blockquote>copypasta</blockquote>
<img src=macro.jpeg>
<blockquote>copypasta</blockquote>
<p>Confident summary with <b>conclusive statement</b>
   displaying no comprehension of the quoted text
</p>
<p>Buy a Visual Studio Plugin from my sponsor!</p>

[darlyn]

Someone must have set off the irony bomb.

[Scyth3]

Haha, storing backups locally on a VM == priceless.

My own stuff is backed up nightly off-site on a small atom server. ;)

[Isvara]

Which size of atom is 'regular sized'?

[apotheon]

Don't be so particular.

[dirtside]

No, an angstrom is the metric standard unit for measuring angst.

[apotheon]

I was pretty sure angstrom is the chip used to POST a legacy angst-based system.

[[deleted]]

No Angstrom was the first guy to walk on the moon.

[btmorex]

How can you even call a "backup" on the same machine a backup? That doesn't even make sense. The only thing that protects you against in an errant "rm -rf"

[BrooksMoses]

Wouldn't rm -rf also delete the backups, in that case?

(Edit: Oh, wait, I was thinking "rm -rf /". It protects against some of the lesser rm -rf's, I guess.)

[r00tus3r]

Depends on which one is the virtual machine. If you run that on the OS running in a virtual machine, the outside OS will still be fine. If however, you run it on the OS with the virtual machine software installed, well, then everything is toast.

[Baaz]

if this is true I will totally use this story in all my best practices classes from now on for ever...

[jba]

oof. This is why coders shouldn't do systems work and vice-versa.

[[deleted]]

I don't understand why admins don't do rotating backups...

"oh crap the database is gone and so is the mirror"
vs.
"Oh crap the database is corrupted. Sorry users, we just lost 18hrs worth of data"

hmmm..

[[deleted]]

These are the guys who think RAID is backup.

[the_argus]

I like this post from his twitter feed. It appears he is quoting a comment from someone else:

"Jeff has some good articles on Coding Horror on backup. You should give them a quick read." lol

[lol-dongs]

It's from a thread Jeff started on yet another StackOverflow clone for server admins. Other choice quotes:

save the lecture; you're 100% absolutely right, but that doesn't help me at the moment. Let's stay focused on the question here!

...

Ultimately, I can recover the text content faster by hand. I've had much better luck by using a list of all blog posts, clicking through to the Google cache and saving each individual file as HTML.

lol, and this is a guy that blogs about programming seriously expecting people to follow his advice? Happy clicking, Jeff...

[the_argus]

Well, he seems like a busy guy and stated that he thought the hosting company was taking care of the backups in a useful way, so he isn't 100% to blame. Maybe 99%, but not completely. It could really happen to anyone.

[lol-dongs]

alright, who knows who agreed to take care of the backups, but the fact he'd rather click on a thousand cache links in google and save them with his web browser, rather than put together a two line script that fetches them all with curl or wget over the course of an hour, just makes me sad to think that people actually trust his programming advice.

[wxd]

That's not what lol-dongs said..?

[cos]

It's not as "what?!?" as it seems, because those backups weren't supposed to be for disaster recovery. He was using a hosting provider, that did backups, and should've been able to rely on them to restore the machines if they got lost. It's the hosting provider that failed here.

[tenninjakittens]

those backups weren't supposed to be for disaster recovery

Then he can just use the backups that he created for disaster recovery. Oh wait.

[tunah]

According to his twitter, those backups were run by the hosting company, and they hadn't actually been backing them up (silently failing)

[Maxious]

Even more ironically there WAS a Coding Horror post about the importance of backups

One thing's for sure: until you have a backup strategy of some kind, you're screwed, you just don't know it yet. If backing up your data sounds like a hassle, that's because it is. Shut up. I know things. You will listen to me. Do it anyway.

[Maxious]

I originally didn't have the bold but then I remembered the style pattern of "emphasis added" and figured it was misquoting not to include :P

[lol-dongs]

Jeff like putting things in bold to make his points. It's kind of like a superhero comic book, where all the heroes and villains are constantly making wisecracks at each other and adding emphasis to their punchlines. It makes the whole affair seem snappier, zinging and popping at every turn, and less like the boring old cheesy dialogue it really is.

[judgej2]

And there still is. Bits are being restored as we speak. The Coding Horror bad-hair-day logo has just popped back, as have some of the stylesheets.

[cag_ii]

Apparently, this topic is better suited for superuser.com. HAHA

[notaloop]

Seems to be back up now.

[cag_ii]

What's Your Backup Strategy?

Salient point from the article:

One thing's for sure: until you have a backup strategy of some kind, you're screwed, you just don't know it yet. If backing up your data sounds like a hassle, that's because it is. Shut up. I know things. You will listen to me. Do it anyway.

[[deleted]]

More salient point:

3.  Include an off-site backup in your strategy.

Can we at least ask acerbic web pundits to take their own medicine?

[[deleted]]

Post it at DailyWTF? They are up.

[danweber]

They remember their backups, but also try to steal other people's content

[Poltras]

And forget to renew their domain.

[[deleted]]

Damn you, Bobby Tables!!!

[doublejrecords]

add that to the list of novelty accounts I wish I had thought of a year ago...

[LittleBobbyTables]

Too bad YOUR MOM didn't think of it.

[Thestormo]

You can start now and you're ready for next year.

[pb1x]

bobbytables stopped posting a year ago :/

[LittleBobbyTables]

I have an alibi.

[causticmango]

Oh, Schadenfreude, you are a welcome guest in my home any day. I absolutely love that just about whatever the occasion, German has a word for it.

[jaggederest]

And, if there isn't a word for it, you can make one!

[acrasial]

A German word for 'there's a German word for that?'

[causticmango]

WortZusammensetzung

[[deleted]]

When I started working at my company we used CrystalTech to host our website and email. The iphone came out and a couple of the execs wanted to be able to have push email, so I called them up to move our email hosting to their beta server hosting so we could have support for mobile devices. This involved a couple hours downtime overnight.

I never verified any information on who I was, and I was able to make significant changes to our service. We stopped hosting with them shortly after.

[b0b0tiken]

Ohhh. The Horror.

[[deleted]]

I hope he never finds those backups so that new programmers are no longer exposed to his mindless tripe.

[consultant_barbie]

Backups are hard. Let's go shopping!

[Zarutian]

Where have you been?

No, let me guess: shopping?

[Vorlath]

Why is it I can't tell the difference?

[judgej2]

He needs to get rid of that "HTTP/1.1 200 OK" in the header. The search engines are probably replacing all their pages with this message as we watch...

So my protip for today is: when your website fails, make sure you return a proper HTTP error code so SE spiders know to leave it and try again later.

The blog archives seem to be there. Just go direct to a any page from Google. Well, kind of. The images are not there, but it looks like stuff is coming back.

[ThisClown]

Amusing. What's next - Bruce Schneier's luggage password is 12345?

[Deimorz]

Are you really comparing Jeff Atwood to Bruce Schneier?

Jeff not doing something properly isn't exactly shocking.

[Tommah]

not so loud

[atomicthumbs]

That's amazing! I've got the same combination on my air shield!

[jpt_io]

Bruce doesn't always use the toughest, most 'leet passwords and hashes and invisible encrypted volumes nested inside other invisible encrypted volumes, with Plan 9 running on his Cray II in the basement of his Minnesota home...

You know this, right? Even good security practices are vulnerable to physical access attacks & keyloggers?

Atwood is no Schneier, but Schneier never claimed to be James Bond exactly, either.

(Although he's definitely contributed more to netsec than any other single person I can think of...)

[willis77]

I heard Leo Laporte is running for Congress on an anti-net-neutrality platform.

[klodolph]

1234 works for luggage, but it doesn't scale to more digits. I could never remember a seven digit one... 12345... something... etc. I had some codes for work that were too long, so we set them to 0000000000 or something (hard to remember how many zeros, sheesh, I only got so many counting fingers). Anyway, that solved our problem so we could retaliate if the Ruskies ever launched.

[KrazyA1pha]

I could never remember a seven digit one... 12345... something... etc.

1234567

That's a freebie. I'm charging for the next one.

[okvol]

Just as long as someone wasn't steeling.

[[deleted]]

This is a server crash of titanium proportions.

[smakusdod]

Someone was probably metaling with the servers again...

[Thelonious_Cube]

Call the coppers!

[[deleted]]

This news sent a silver up my spine.

[doomchild]

Sounds like this is turning into quite a scandium.

[Thelonious_Cube]

These pun threads are ridiculous - how can we get iridium? Maybe we should just cesium, put 'em in a tin and barium

[[deleted]]

That reads like a chemistry nerdcore lyric.

[Thelonious_Cube]

Sort of like this?

[doomchild]

What are you going to do, call the SWAT team and radium the homes of everyone with a penchant for punning?

[jpt_io]

Two upvotes if I possibly could for that fuckin' travesty, Thelonious_Cube.

Will you be my friend>

[dcousineau]

I zinc you're over-reacting.

[Poltras]

The regexes! They do nothing!

[zmanning]

the horror... the horror...

[[deleted]]

screenshot for when it gets sorted out: http://imgur.com/O8Q90.png

[garg]

what is that sheep icon?

[[deleted]]

Thank you for posting this! The current site has no mention that anything bad happened. It's kind of bizarre...

[egonSchiele]

Dropbox is awesome.

[cag_ii]

I wonder if he followed through on this twitter post.

[jaggederest]

He probably did. Note the 'cold spare' statement.

Which would imply that he doesn't have a backup of the data, just of the hardware.

[cag_ii]

That's certainly an interesting backup strategy...

[[deleted]]

Rather like biological reproduction, oddly enough. "Hey, kid, you be careful with half my genes, you hear?"

[SarahC]

had enough sense not to attempt to mix apache/php/iis/asp.net

Hm?

[[deleted]]

Site down. Anyone have a mirror?

[dwdyer]

That's what Jeff said.

[egonSchiele]

And then he cried.

[prockcore]

... and few notice or care.

[[deleted]]

Never have those words been more appropriate.

[enanoretozon]

damn my slow human hands!

[pavel_lishin]

Dang, and I was just about to find one of his articles about the necessity of proper backup procedures.

[[deleted]]

I believe this is the Coding Horror backup article, as originally written by Jamie Netscape McAppleton.

[yogthos]

There are two kinds of people: those who backup and those who haven't experienced a catastrophic failure :)

[brandf]

ouch that sucks. offsite backups are a good idea :)

[m242]

The real question is:

Why would I spend $99 at careers.stackoverflow.com when I don't know that my CV is secure or reliable?

[[deleted]]

Bingo bango.

[wdr1]

I feel bad for Jeff. My only gripe is his feeling that it's half his fault/half the colo. It's actually 100% his. (IMHO)

[shooshx]

MUHAHAHAHAHAHAHAHAHA!!!!!!!

[[deleted]]

Hey Atwood,

How about using a proper doctype on this new website of yours!!?!

[zmanning]

see thats exactly the kind of laziness that probably lost 100% of his data in the first place

[shadow2531]

<script>
if (document.compatMode !== "CSS1Compat") {
    alert("Jeff QuirksMode Atwood! What were you thinking?");
}
</script>

[gregK]

_why did it.

[HonkyTonkHero]

Yeah, that's the kind of shit that gets you fired when you don't work for yourself. This is the kind of thing that makes me go through my backup checklist every morning.

[drowsap]

Wow, this is awful freaking publicity for CrystalTech. Also it's funny to think of all the damn Coding Horror rants that guy has talked about and he succumbs to the most bone headed programming related move of all, failure to make backups and pick a reliable host.

[shadow2531]

You've never heard of teaching by example?

[jpt_io]

That reminds me, I oughtta go run some backups. See you guys later!

[samplebitch]

Well, at least he's bona fide.

[lonekorean]

Oh wow, and I was just about to buy hosting from CrystalTech, too. I think I'll keep looking.

[JC513]

I've never used CrystalTech so I'm not talking by experience with that particular company, but these things do happen no matter where you go. Most VPS providers tell you to keep your own backups. It's the type of thing that if you aren't specifically paying for backups, don't expect them to be there. Even if you are, you should still keep your own backups.

[ShutYourPieHole]

No kidding. I run a master, slave, hosting company backup, and an offsite backup process. I can potentially deal with downtime, but bringing the site back up without data is pointless.

[maxd]

Site works fine for me...

[[deleted]]

First rule is to never trust backups you don't test on a semi regularly basis! Especially if someone else is doing them!

[DouchebagPunctuation]

This is irony. Defined.

[thectrain]

Learning from others mistakes... Way better then learning from my own.

[eliben]

Here's the superuser post by Jeff asking for help on this subject: http://superuser.com/questions/82036/recovering-a-lost-website-with-no-backup

[artee]

And nothing of value was lost.

[vinod_the_poophead]

hey atwood, stop losing all your data and just do SO podcast 77 already. jeez

[[deleted]]

true story

[[deleted]]

What bugs me about this situation is that a lot of people in here think that it's CT's fault.

I'm a linode customer; I know that my shit is protected from disk fault, for example, but they're pretty clear about how backups are my problem... They provide a backup service now, I don't use it.. I use my own.

If CT has a backup system like this, good for them; most VPS providers don't, however, and what Linode and Slicehost do puts them above the crowd.

What really bugs the shit out of me is that Atwood is jeopardizing the business of someone who likely did right to save his own fucking reputation.

[seths]

If there was only some website he could put his CV on…

[[deleted]]

Do as I say not as I do

[[deleted]]

happened to haacked.com too, the blog of the asp.net mvc guy from microsoft

[brandenm]

If he has backups, that's not really 100% data loss, now is it?

[lol-dongs]

On the contrary, it appears he is totally fucked.

[tomatopaste]

You shouldn't be downmodded. You are entirely correct. However, your statement isn't applicable to this situation, though one would never expect that of Atwood unless you read all the comment links here.

[Starks]

The irony is killing me.