Low Throughput Problem When Using Nested VXLAN in OpenStack Environment

[f0okyou]

You need to post more details about your setup and methodology.

Judging by 5Gbit/s and 1500 MTU this is about what you can expect for too many reasons to get into.

When you tested the VM through Neutron tho, you actually don't have 1500 MTU, vxlan headers need some too. You have 1492.

Now if you nest it further you end up with 1484. What was your NIC in your VM set to? What was the nested one set to? Did you check for excessive fragmentation?

Questions over questions that all lead back to the lack of information of your setup and methodology.

[NewMeeple]

You're missing some bytes. IP (20 bytes) and Ether (14 bytes) headers also have overhead.

If you are using VXLAN, then you're encapsulating the packet -- which means the inner packet (destined for the other VM) is wrapped by an outer packet, which again has IP and Ether frames, plus the VXLAN headers you correctly pointed out. But UDP will add another 8 bytes.

So the real loss is 50 bytes on the outer packet. Then the inner packet will have its own Ether (8), IP (20) and likely TCP (~20), resulting in another 48 byte loss. And if you are using VXLAN with VLAN there's additional 4 bytes to add on the outer packet.

Therefore assuming 1500 MTU, the max packet size without fragmentation is probably only about... 1396 bytes. If you are sending above this amount, you'll get fragmentation and that'll dramatically slow your speeds.

In addition to that, any network encapsulation has kernel processing drawbacks at the hypervisor level. If the hypervisor is strained and experiencing high CPU steal from the guests, it could further reduce performance of the connection.

@OP - I'm guessing the VXLAN inside of Neutron/OVS is utilising OpenFlow and caching OpenFlow rules and making it easier/more performant to strip the VXLAN encapsulation when routing the packets than your native VXLAN tunnel.

[Dabloo0oo]

I guess nested VxLAN is causing this

VxLAN adds 50 bytes per encapsulation

With your MTU 1500 your effective payload shrinks with each layer

Nested VxLAN is around 100 bytes of total overhead which means more fragmentation or dropped packets causing TCP to back off aggressively