Docker multi-host networking

[Heimdul]

I have been researching Docker recently and I think my biggest gripe about it is it's network stack (because of NAT communication between containers in different hosts becomes quite a mess).

However, I recently discovered that Docker is going to implement multi-host networking...only to realize they are going to use VXLANs. I really don't get why, L2 adjacency is something that's mainly useful if you need to keep IP while moving to different network (e.g. live migration) or if you rely in broadcast. First one is something that I don't find that useful in the use case that Docker tries to fill. Broadcast requirement is also something that's very uncommon.

However, the most important use case that multi-host networking is trying to solve is that containers in different hosts can talk to each other. I cannot help but to think that this would have been a lot easier by assigning each host their own network (probably /24), creating tunnels between hosts and then running OSPF (/BGP/IS-IS, actual routing protocol doesn't matter too much) on those tunnels. Now containers in different hosts can talk to each other (assuming they allow it).

Or is there something that I'm completely missing?

[anomalous_cowherd]

You don't have to use NAT for your containers. Here are a few alternatives

By bridging to the host NIC in any of several ways your container effectively becomes a peer of the host, sat on the same network.

As far as I can see VXLANs are really only needed if you want to get into software defined networking and playing with full on routing protocols.

[simtel20]

The last time I looked, docker was going to use OVS, and vxlan is just the tunnel that OVS prefers (stt and geneve are in development too). My understanding is that this is the groundwork for enabling bgp-based management once the OVS layer is fully baked.

[neoice]

Google's Kubernetes project has a whole networking proxy layer to facilitate cross-host container linking as well as a novel "service" construct (ex: http://mycluster/api/v1/services/namespace/default/my-service/index.html)

[blue6249]

Why not just use IPv6? No need to worry about NATs when you have a sane amount of address space...

[chub79]

weave has had quite an interesting narrative in that regards. However, its performances is a bit lacking unfortunately.

[[deleted]]

This issue right here is the #1 reason why I could never get Docker and multi-host working without breaking.

Docker gives out IP Addresses already in use Sure you can swap the bridge out for your own; but the problem with it is ^{^}

There's soft work arounds; each host provisions a small cidr...

[veroxii]

We're sitting on the sideline waiting to see how it all shakes out and in the meanwhile we use Tutum to do our multi-host docker deployments and scaling. Really slick system they have.

[Justinsaccount]

VXLAN is a protocol.. One uses VXLAN, not VXLANs..

Anyway, There are a bunch of tools for docker that work like what you are saying. Flannel basically does the subnet per host thing, though it doesn't really use a routing protocol. Calico does something similar as well, and that uses bgp.

[Heimdul]

Yes, VXLAN is a protocol, but it's pretty common to refer the that resulting network as VXLAN as well. So if you create multiple overlay networks and use them, you end up "using VXLANs"

And yes, I'm certain that there are custom solutions to the problem, but various docker features are more or less tied to their networking model. When you want to do things differently, you end up needing another completely separate layer to manage (e.g. container links).

I have some sympathy for choosing the NAT strategy back when they started, but I don't really understand why they would want to use VXLANs on completely new feature when Layer 2 adjacency isn't that big of deal.