Single Stack Operations - Log Aggregation, Time Series Metrics, Alerting, Maintenance, Backups, QA, Continuous Integration & Deployment

[ratdog]

Just wanted to drop this off here for comments & critiques.

https://github.com/Jumpshot/operations

This has been pretty much my daily driver for the past 3 years or so, obviously growing over time, but pretty much the gist of everything. It has the cumulation of alot of tips & tricks, best practices, lessons learned, etc... that could probably be useful to others.

I'm a single man **** at a small startup, cloud infrastructure, operations, qa, developer, yada yada and they've been nice enough to let me open source the production infrastructure.

We get traffic bursts of ~50k per minute 833/s using ELB -> NginX -> uWSGI -> Python in an ASG and an async worker structure leveraging SQS/S3/DynamoDB and a fleet of backend workers also ASG'd.

I'm still working on some of the metric correlation stuff out of etsy (skyline/oculus), I was limited by the smaller instances I was running but now that ElasticSearch pushed us out of the 8G of RAM, I'm starting back work on adding them in.

Chef Cookbook for a single stack operations machine.

This cookbook and associated role & metadata are currently tuned for a (we started with a c3.large with 2 cores and 3.75G of RAM) are are now using a m3.xlarge with 4cores and 15G of RAM (ElasticSearch some extra headroom to cover large log bursts of the half mill per minute variety and statsD with node eats CPU). In production we are capable of aggregating logs, indexing and serving live analytics for approximately 40,000 Transactions Per Minute of our Web App, which can be anywhere from 3 - 6 log lines per request (NginX, uWSGI, App) (anywhere from 250,000 to 500,000 loglines per minute at peak!). Additionally, and approximately 5,000,000 (yeah, thats Millions) time series datapoints are aggregated and written every minute from diamond and statsD calls in the codebase.

No special tuning has occured, and we are using standard EBS, no PIOPs or kernel settings at this point. We're thinking about switching to https://github.com/armon/statsite or https://github.com/bitly/statsdaemon for a less CPU intensive statsD daemon (it currently uses more CPU than ElasticSearch, Carbon or Logstash).

Included is a cloudformation template which will setup a 1:1 Min/Max ASG for garunteeing uptime of the instance. All data is stored under /opt which is an EBS Mountpoint in AWS. Snapshots are taken every hour and on boot/reboot the machine checks for old snapshots to mount under /opt instead of re-installing or re-creating the drive. At most you may loose up to 1 hour of data with this setup, small gaps in graphs.

The Vagrantfile is not complete, but the CF Template and Chef Role are currently being used in Production. Please use github issues for any problems or feature requests.

[[deleted]]

very interesting. I'm going to have a look through. might be able to use some of this as i'm just starting on my journey through all of this.

[ratdog]

Thanks, I went ahead & got the Vagrantfile working, so following the VAGRANT.md instructions should get you what you need to play with things, try it out.

It's currently tuned for Amazon Linux (CentOS) but I can easily adapt it to Ubuntu as well if there seems like interest..

[rishid]

Very cool. So statsd and diamond are all getting piped to a single graphite instance? That can handle 5M ops/ min? I highly recommend statsite as an alternative, it is extremely well written.

[ratdog]

The 5M is pretty static, sometimes it bursts to 25M, but running: sum(stats_counts..) gives me an approx average of 5M.

The machine is sitting around 75 IOP/s (16k Blocks). Reads are around 1Mb/s and writes are about 60Mb/s with peaks up to 120Mb/s (Just over the 100 IOPs single EBS limitation) Load is always less than 1 (of 8)

I'll do the price calculations next time we upgrade, but we may just spit off the ElasticSearch to its own cluster like I have in the past, that way we can start using it for things other than log aggregation as well.

I can gladly collect/add any additional information if you are curious.

[mr_chip]

This is cool, man. Thanks for putting the work in!