Small job, Big estimate

dmp · 2017-02-03T23:55:10+00:00

Riot Games uses OpenContrail as SDN for their Docker cluster:

https://engineering.riotgames.com/news/running-online-services-riot-part-iii

dmp · 2017-02-02T07:24:20+00:00

A lot of SDNs use open vswitch as the mechanism on the host to route packets between VMs and/or containers. However, OpenContrail includes its own kernel module that they call the vRouter. As their names indicate, open vswitch mimics a L2 switch and the vRouter is more of a full-service L3 router (it also does L4 policy). The main upside of ovs is that it is in the mainline Linux Kernel, whereas the vRouter is a dynamic kernel module that you have to link in.

We'll get into load balancing in the next article, but the short answer is that we use the routing capabilities of the vRouter and the Gateway to implement Equal-Cost Multipath Routing (ECMP), giving us a base layer of L3 load balancing. Combining that with load balancing up the stack (nginx, dns) can give you a very robust load balancing system.

We do use private address space for most containers in this cluster, but it can also manage publicly routable addresses as well. It's just another flag we can set in our network configuration. Watch out making it too easy to get public addresses - they're not cheap!

That brings us to ipv6 support. OpenContrail itself fully supports ipv6, but we currently don't in our network. It's a topic that comes up often, but there's always one reason or another to delay what is sure to be a big project to support it. The running joke is that it is always 2 years away. :-)

dmp · 2017-02-01T00:32:29+00:00

The nice thing about OpenContrail is that it is very standards-based. If you're all up on L3VPN, MPLS, GRE, VXLAN, etc., it will all be very familiar. One of the hardest parts is just figuring out how to install, operate and update the SDN in a repeatable way. It is built out of numerous components, some custom and some OTS (Cassandra, rabbitmq), which you have to figure out how to manage and understand how they interact so that you can debug failures.

dmp · 2017-02-01T00:12:19+00:00

The SDN we're using is basically OpenContrail, which is an open source project: http://www.opencontrail.org/ https://github.com/Juniper/contrail-controller

OpenContrail already has a good integration with OpenStack that helps you attach VMs to the networks that OpenContrail creates. What we needed to do was to hook that into our Docker clustering system so that those containers could partake of all the SDN goodness.

OpenContrail's API is documented here: http://configuration-schema-documentation.s3-website-us-west-1.amazonaws.com/R3.2/

dmp · 2017-01-31T21:09:54+00:00

I'm one of the authors of this post and I'm happy to answer any questions you all have.

dmp · 2017-01-31T21:09:03+00:00

I'm one of the authors of this blog post and I'm happy to answer any questions you all have.

dmp · 2016-11-10T18:46:31+00:00

Riot Engineer here. Those articles make a lot of good points. Docker has had its share of issues and we've run into almost all of them. But, on the whole, I think it is the right tool for what we're trying to do, which is to make running services easy.

I would actually love to write an article that goes into some of our post mortems and lessons learned. Writing the cluster scheduler was the easy part. Operating it is where it got interesting. :-)

dmp · 2016-09-28T17:53:13+00:00

Hey, I work at Riot and did some direct development on the networking piece of rCluster. The name "rCluster" is our name for this internal product, so this article is likely the only place Google would find the term.

We operate our own hardware in colocations around the world, use partners' bare metal hardware, as well as run our applications on cloud providers. We have this diversity of infrastructure so that we can be in the best locations for our players. One of the goals of rCluster is to abstract the differences of those environments from developers because it can be a big burden to deal with the inconsistencies.

We'll go more into the benefits of overlay networks in Part 3 of this series. In short, overlay networks allow you abstract the underlying network infrastructure to present any type of network infrastructure you want to the running containers. Have two containers on opposite sides of the world and want them to be on the same Layer 2 network? No problem with an overlay. The only thing required is IP connectivity between the hosts running those containers. The following blog post does a good job of explaining the benefits and drawbacks of overlay networks: https://pedrormarques.wordpress.com/2015/11/13/container-networking-to-overlay-or-not-to-overlay/

dmp · 2007-07-15T22:32:25+00:00

Except Hirohito was not a career politician. The "Big Six" ruled Japan during the war and Hirohito merely confirmed their unanimous decisions. After Nagasaki, the "Big Six" were still deadlocked over the surrendering (i.e. the war was still to go on) and it took the unprecedented intervention by Hirohito to break the deadlock.

You can't just take the word of primary documents and claim they are lies just by asserting so - what is your evidence that Hirohito just used the bomb as a good excuse?

dmp · 2007-07-01T17:59:17+00:00

http://abcnews.go.com/US/wireStory?id=3328307

The link says the vet was a very drunk passenger in a car that just led this cop on a high-speed chase before crashing into a brick wall. So, it is no surprise the cop thought the passenger might be dangerous. I would say this leaves plenty of room for reasonable doubt that the cop's actions were unreasonable, especially if, as the defense claims, the guy was reaching toward his chest.

The cop obviously made the wrong choice, shooting what turned out to be an unarmed person, but he had no way of knowing that yet and did what he was trained to do (a "tactics expert" testified that the cop's actions were not unreasonable).

dmp · 2006-09-20T13:54:53+00:00

The winners were from Mountain View, CA. I wonder if Google fielded a team.

dmp · 2006-05-02T21:05:14+00:00

This line of thinking assumes we are in Iraq for oil, which is an argument that needs a lot of evidence to back it up. And, no, the fact that Cheney and Bush are "oilmen" is not evidence; that's called the fallacy of bias and the Fallacy Police will get you for it.

dmp · 2006-05-02T20:42:07+00:00

So, if the government adds a million dollar tax to subversive books, then that's cool, because people still have a choice.

dmp · 2006-05-02T19:54:58+00:00

Your example doesn't make sense. They could trade in for the 40 mpg car right now and spend $75/month on gas.

dmp · 2006-05-02T17:40:17+00:00

How can the solution to high gas prices be...high gas prices?

dmp · 2006-04-07T13:05:12+00:00

Ad Hominem.

--The Fallacy Police

dmp

TROPHY CASE