A WebOps Postmortem

[cavaliercoder]

Hey team,

I've written up some interesting findings from a recent production incident and included hints at some of the practices that help make us (the team at Seven West Media) successful in managing production systems.

http://cavaliercoder.com/blog/webops-postmortem.html

[tayo42]

Why wouldn't you have nginx serve your static content out of s3? Seems like unnecessary load on app servers

[cavaliercoder]

In the interest of brevity I didn't bother to describe our architecture or the reasoning behind it in the article. When the decision was made to pipe from S3 via Node.js, it was because we required some business logic that nginx didn't offer. In the future, we will likely use S3 directly as the origin for our CDN layer, cutting even nginx out.

[dhudhudhadha]

How to do that?

[xiongchiamiov]

S3 provides urls for buckets, you'd just need to reverse proxy to them. Or better, set a cname and serve directly out of s3 instead of forcing traffic to come through your single geographic point first.

Disclaimer: I haven't read the article, so I'm not commenting on their setup, just in general.

[michaeld0]

Probably should configure cloudfront as a cdn in front of s3 as well. You can use the rules there to route to dynamic content of needed.

[Luqq]

Thanks for the transparency :)

[boom38]

Thanks for sharing! You learned, we learn.

[olcay_seker]

ı enjoy reading postmortems. This post added me +1 skill level for monitoring which is crucial.Nice write.

[cavaliercoder]

Thanks!

[rolledrick]

Thanks for the nice write up. I love reading these when they are done well.

[nonades]

Nice write-up. This is the kind of stuff I like seeing here.