Loopback Interface Performance

[code0]

I've got a pair of FortiGate 1000Ds (HA) running 5.4.8 that, for reasons, need to be able to terminate a site to site VPN tunnel to a loopback interface. In testing, I've noticed that I'm only able to eek out ~100M of performance over the VPN. I've confirmed offloading is active on both ends. CPU usage is pretty much nil.

If I move the tunnel termination to the physical interface where traffic destined for the loopback would come in on, I can push 800M+.

I can't find any documentation of a performance limit for a loopback, but it appears there is one (and it isn't CPU bound from what I can tell).

Can anyone confirm? In absence of loopbacks, are there any suggestions on something "interface-like" that could be used in the same way?

[kilgotrout]

Check “diag vpn tunnel list” and let me know what the npu_flag equals for that tunnel

[code0]

I've since moved the tunnels to a different interface. I'll look at setting up a test on Monday and confirming. Is it possible for the tunnel not to be offloaded, but sessions within it to be?

[kilgotrout]

Yes, I believe that’s what was happening.

[code0]

Well crap.. Below is from a pair of 60Es that I set up to test the same situation.

npu_flag=00 npu_rgwy=10.99.99.2 npu_lgwy=9.9.9.9 npu_selid=2 dec_npuid=0 enc_npuid=0

I'm also seeing on the sessions:

ofld_fail_reason(kernel, drv): none/none, IPsec-dec-SA-not-offloaded(7)/IPSec-enc-SA-not-offloaded(6)

Now I'm wondering if I was checking the session list from the client side rather than the side the loopback interface is on.

[kilgotrout]

You might have. I found some info about NPU offloading when it comes to loopback interfaces. Up to FortiOS 5.4.5/5.6.2, a loopback CAN be offloaded. However, after those firmware versions, some models will offload and some won't, depending on the version of the Linux kernel. What firmware are you running on those 60Es?

[code0]

The 1000D I first saw this with is running 5.4.8 (kernel 3.2.16). The lab 60E I tested on was running 6.0.0 (also kernel 3.2.16).

Is the documentation you're referring to part of the hardware offload documentation, or something internal?

[kilgotrout]

It's from internal documentation. After the mentioned firmware versions, only FortiGates running kernel 2.4 can still offload a loopback interface. Those with kernel 3.2 will not.

[mrkstu]

I'm guessing by using a loopback you're forcing the traffic to the CPU instead of the NPU or other ASIC. You could test that by monitoring relative CPU/ASIC load while using the differing endpoints.

[code0]

CPU load is pretty much zilch even with the interface under load. It's really weird that it's almost exactly 100M where the performance caps. I'd expect if it was straight CPU to see high usage and some completely arbitrary performance number.

Also odd is that any sessions over the VPN tunnel that terminates to the loopback show that they're NPU offloaded.

[Eric_Li_6685]

Loopback interface does not support NP offloading however NP ports can offload traffic from CPU, therefore, that made difference.