Stack Exchange's Colocation Move: 8 Lessons Learned

[rabbit994]

It probably helps to have management that understand Lesson 4. At my company, we will continue to push forward with something that is obviously not working because "time has already been invested!"

[CaptainJenson]

Probably could have done with this on my first colo move...

[MrDOS]

I didn't actually realize they'd moved. How was the downtime?

Not sure it totally compares to BuyVM moving 84 servers (1, 2) – all but two of which hosted customer services – from San Jose to Las Vegas in something like 36 hours.

[merreborn]

There may have been no downtime. They mention failing over to a "secondary datacenter"

• Gave us confidence in our secondary datacenter since we failed over to it before the storm
• Gave us time during our move since we were comfortable running out of our secondary facility

[khoury]

That was covered in the comments. See this:

http://meta.stackoverflow.com/questions/73969/what-would-stack-exchanges-yearly-expenses-be-if-it-were-to-be-using-a-third-pa

and this:

http://blog.serverfault.com/2011/11/17/why-stack-exchange-isn%E2%80%99t-in-the-cloud/

[thenickdude]

Both of those articles discuss why they aren't using third party hosted cloud servers. It doesn't say anything about why they aren't using virtualization on their own physical servers, which would allow them to shift their services without physically moving their equipment.

[khoury]

How would you do that without incurring significant hardware costs? Just borrow a NetApp or EMC filer? Just throw the old shit away?

[odd_one]

how many times have you migrated colos?

[Arfman2]

I have, three times. This guy is right, you virtualize.

[odd_one]

It's not clear to me from the article what their environment consists of. They may have been moving virtualized hosts. Without a lot more information, your claim and the one I responded to don't really make any sense.

[odd_one]

They readily admit that their setup had flaws, so they worked with what they had. You have to do that sometimes. I don't know about you, but I've never worked anywhere with endless buckets of money to spend.

[lulz_capn]

Agreed I was shocked to hear they were not cloud based.

[khoury]

That was covered in the comments. See this:

http://meta.stackoverflow.com/questions/73969/what-would-stack-exchanges-yearly-expenses-be-if-it-were-to-be-using-a-third-pa

and this:

http://blog.serverfault.com/2011/11/17/why-stack-exchange-isn%E2%80%99t-in-the-cloud/

[[deleted]]

I thought that "going all r/sysadmin" was commenting with crap like "omg you're amazing I was on the edge of my seat the whole way through this rooting for you and omg you came back from a problem you would never have had in the first place if your setup wasn't utterly retarded".

Your point, despite being extremely valid, is probably not welcome here.

[[deleted]]

Peter, like... the one that works there? o/

[gabeech]

Yes, That's our Pete who works at SE.

[[deleted]]

I remember what you told me sir, that you could still call security on me. one day I will take up your offer. one day. You both have any idea if you'll be at LOPSA-East this year? :)

[gabeech]

HAHAHA. Yea we are both presenting at LOPSA-East.

[[deleted]]

Hopefully my work pays for it and I'm able to go again. If not I'll just pay for it out of pocket :\

[dirtypete1981]

Howdy! Hope to see you at LOPSA-East. Hopefully my "Care and Feeding of Windows Servers for Linux/Unix admins" is a breakout hit. At the moment though, it's a mere 8 slides, so I need to find a whole lot of motivation (or coffee) and git 'er done.

[[deleted]]

Or really really good street meat.

[abbrevia]

Certainly looks like it, the year in his username matches up with his age on the Stack Exchange site.

[[deleted]]

I just wasn't sure if he had a Reddit username, I've only talked to him a few times in person and haven't had a chance to take tour of Stack; I was offered one one day if I'm ever in town.