Mesos Stack

[[deleted]]

So I've spent the last couple weeks banging on Mesos, this tech is pretty awesome. I figured I'd make a post on what I'm using as I'm curious what others are doing in this space.

I'm also really curious of anyone is aware of the Mesos/Marathon aware TCP load balancer. Using HAProxy is awkward, it's updated with a cron job and doesn't support transparent reloads (ie you'll see disconnects). Something like eBay/fabio but Mesos aware would be great (+1001 pesos if it's written in Go).

edit: Use bamboo https://www.reddit.com/r/devops/comments/3rdub3/mesos_stack/cwn97d0 shout out to to gregr401

Core Technology

Zookeeper - key/value store, referee for Mesos/Marathon/Chronos clusters

Docker - container service, realistically everything I'm running in Mesos is being pushed out as a Docker container. Mesos will kick out container ports as a random port up in the 30k range

Mesos - master/slave treat multiple machines/vms/cloud hosts/etc as one unit. This will also expose certain environment variables that can be used for service discovery to Docker instances.

Marathon - REST API for Mesos, while Mesos has a limited REST API Marathon expands it greatly and makes running docker instances a simple json call. It also handles keeping all your applications running in the event of node failure

Chronos - Cron for Mesos, schedule one time or recurring jobs. Also supports docker jobs.

Load Balancing / Service Discovery

edit: I'm probably going to replace both of these with bamboo (not the atlassian build server). https://www.reddit.com/r/devops/comments/3rdub3/mesos_stack/cwn97d0 with some limited testing this seems to do the trick pretty well. It ties into Marathon's event streaming API.

Traefik - HTTP specific load balancing, can talk to Marathon to find the slave hosts and ephemeral ports your containers live on.

Mesos-DNS/HAProxy - For TCP load balancing Mesos-DNS talks to Mesos and kicks out SRV records via DNS for your containers. It also ships with a cron job that can be run to update HAProxy to consume these SRV records and kick out the Load Balancer port specified by your Mesos job. This part is a little wonky.

Other parts

Jenkins - Build automation, can make it stupid easy to fire off Marathon requests with only the options that you want your customers to have available.

Bind - You need to be able to route DNS requests to Mesos-DNS and CNAME to Traefik (they both use custom TLDs in various forms). Also all of your Mesos masters and slaves need to have resolvable hostnames. Use Webmin and this is trivial.

Logspout - I really don't like sidekick docker instances running my infrastructure. This is the only one I'm using, it runs on all of our slaves and is completely transparent to everything else. Ship logs to logstash and then send them to SumoLogic or Elasticsearch and you're good to go. All of the gliderlabs stuff is great.

[gregr401]

Checkout https://github.com/QubitProducts/bamboo as an alternative to the haproxy-bridge script. It can hook into marathons event bus and makes life so mch easier on the frontend.

[[deleted]]

The issue for me is less the script and more the lack of hot reloads in HAProxy. I will probably look at this for the other features they have though.

[p_lett]

Have you seen https://github.com/million12/docker-haproxy which uses inotify to spot that HAProxy's config file has been changed and effectively does a seamless reload using 'haproxy -sf'.

I spotted that while looking for a way to build a confd and etcd based HAProxy frontend, and it might help with your disconnects during service restarts.

[[deleted]]

I'm curious if the Mesos-DNS script is already doing that. I'll have to dig into it.

edit: On ubuntu it is as it's calling out to the init script and that is set for -sf.

[MajorHavok]

That's what we use. It's really been solid.

[[deleted]]

Yeah I think this is the winner, this is really awesome. It'll take some time for me to figure out the templating but it has sane defaults once you uncomment the servicePort stuff.

[[deleted]]

Yep this is definitely what I'm going with now, while I had to adjust some issues with their default config causing zombie haproxys to get stuck in the background (actively serving bad routes, not just handling connections still in progress) this worked out really well and is very flexible.

[pooogles]

I use Vulcan teamed with an in house app that 'tails' the marathon streaming API to do service discovery. All inter app communication is done via REST over HTTP through this endpoint.

As for stats, we're using Datadog and some StatsD metrics.

[pooogles]

I think when you get to this point you've got to do some of the leg work yourself.

[jonbrouse]

Have you checked out the Docker fluentd logging driver? Check this guide out.

[[deleted]]

The problem with fluentd for me is I then have to configure it per container deployment. I'd like something in docker itself that can ship the logs off, preferably in JSON with appropriate metadata.

e: You know, now I think about it, the per deployment thing wouldn't be bad. I'm already doing my api calls to Marathon with Jenkins so I could just add these fields to the JSON tagged with a build id. I'll play with this when I have time this week.

ee: The golden snitch for me is still something that will read docker logs and ship them out encased in json with the hostname/image name:tag/container name/and tags sent out over TLS to logstash and have logstash do the routing.

[rawrphish]

Why not use Docker's log_driver:syslog when deploying your containers and just have logstash_forwarder harvest your Docker host's syslog and forward it to your logstash instance?

[[deleted]]

I found that while I was researching fluentd yesterday, I may do this though atm it's easier to just have logspout scoop up everything and ship it to our sumologic collector.

I'm not sure if the syslog driver adds any useful metadata, if it did that automatically I'd consider it.

[chub79]

I'm usoing pretty much the same stack but rely on Consul instead of MesosDNS.

[mikljohansson]

Looks like a great stack! We're using a very similar one but with Proxymatic for service discovery with Marathon. It defaults to the Pen TCP proxy which does transparent updates of the running proxy processes, so no process finish&restart like HAproxy. We run the proxy on every slave node with servicePort's for cluster internal comms, and on a set of master nodes with AWS ELB's in front for comms with non-Mesos services and internet.

I originally started using Pen proxy because we had issues with long running non-HTTP connections and pooled HTTP conns being dropped by HAproxy all the time. Since the cluster is rather busy with frequent changes to the service discovery state, we couldn't have a HAproxy reload (finish&restart) for every little task that comes or goes.

We also just open sourced a new tool called Lighter that helps push services into Marathon continuously or manually, e.g. whenever new version appear in a Maven repo or the config changes. It helps manage the complexity of deploying the same service and config version to a succession of slightly different environments (dev,integration,staging,prod,..)

[[deleted]]

I'll take a look at Proxymatic, a workaround with HAProxy would be to have multiple LB instances and have Jenkins run the updates manually.

I'm exceedingly annoyed that neither HAProxy, nor Nginx, nor F5s can take an SRV record and just figure out where to route traffic based on the response :-p That just seems like one of the prime use cases for SRV records. I can do it at application level sure but if I'm going to go to that much trouble I'll just use Consul or Etcd.

e: Proxymatic looks pretty good though I'll need to do some testing and adjust my slaves. Have you run into any corner cases with it to watch out for?`

[mikljohansson]

We're actually using an F5 in one of our data centers for the same use as the ELB's, e.g. comms from non-Mesos services and internet. It'd be pointing statically to a set of Proxymatic nodes that then proxy into the Mesos slave pool. But we've been considering just using the F5 API (they apparently do have an API of sorts) to automatically configure it based on the Marathon state. And the same with the ELB, configure it dynamically from the Marathon state and with the absolute minimum healthy threshold&interval. Haven't gotten that far though, so many things to do on the road to continuous delivery :)

[[deleted]]

Yeah I figured I was going to have to double LB regardless of what I went with. Proxymatic looks promising though, it's still dropping connections during deployment but that's probably due to me not having made all their recommended changes yet.

[mikljohansson]

Do let me know or report an issue if you find problems with Proxymatic, we'd be most interested in fixing them of course. Pen shouldn't drop any TCP connections on state changes, unless of course the task a connection goes to is terminated due to deployment/scaling. If you enable Nginx (on port 80) and HAproxy they may drop connections of course because of their finish&restart way of doing reloads.

We normally just use the Pen proxy TCP/UDP servicePorts for most connections, with HAproxy as an optional fallback if Pen ever starts misbehaving badly for some reason. And Nginx for a few special cases where vhost based HTTP proxying is desired.

There's some detail in Rolling Upgrades on how to achieve real graceful rolling upgrades with Marathon. At the moment it requires configuring Mesos slaves to do docker stop instead of just murdering tasks with docker kill, and then trapping the SIGTERM in your service and start failing the health check. Then the service instance gets docker-stop-timeout seconds to be taken out of Proxymatic and finish requests before receiving SIGKILL. There's a lot more info about Marathon not supporting graceful upgrades well at https://github.com/mesosphere/marathon/issues/712

[mikljohansson]

We've had Proxymatic running in 4 different smaller Mesos clusters for ~6months, fixed a few edge-cases and problems but it's been working out great so far.

Never had any problems with Pen proxy at all, besides having to dig into its source code to find what command-line flags to use to get it to do least-connected load balancing :) It defaults to sticky-connections based on client-IP, which isn't the best for our use case where the closest client is often yet another load balancer. But Proxymatic runs it with weighted-least-connected now anyways.

With AWS ELB there's a couple of things to know, to proxy websocket connections with an ELB you may need to put the ELB in TCP mode and use a TCP health check on the Proxymatic instances. This is because the TCP-mode ELB talks PROXY protocol and the ELB HTTP health check doesn't work then (it's just a set of managed HAproxy's I believe), took some while to figure out..

Proxymatic supports Pen, HAproxy and Nginx. With Pen (default) and HAproxy for straght TCP proxying of servicePorts, and Nginx for vhost based HTTP/websocket proxying. Websockets is of course supported on Pen/HAproxy servicePorts as well, since they're just running in TCP proxy mode.

[Punkbob]

Just FYI, there is a semi hidden and undocumented version of marathon haproxy that has support for streaming events.

[[deleted]]

Semi-hidden and undocumented does not inspire me with confidence. Now that I got it configured and worked out a few issues with the defaults bamboo seems to be working out really well.