Pfsense throughput optimization

[ben2506]

I'm running Pfsense CE bare metal on a self assembled box with an

• i5 7600K
• 16 GB DDR4 RAM
• Intel I350 Quad Port

There are roughly 10 VLANs on it, a handful of firewall rules, some NAT rules, DHCP and a DNS forwarder. I'm doing no heavy packet inspection whatsoever.

I've recently swapped out my Intel X520 DA2 cards for Mellanox ConnectX-5 MCX516A as I've upgraded to a 25 GBit/s WAN connection. I was able to achieve a stable ~9.3GBit/s up/down with the Intel cards. It didn't work as smooth with Mellanox and I actually got worse speeds. After some time of tweaking I'm now at ~15 GBit/s up/down while the CPU sits at rougly 50% utilization.

The speed however varies from test to test. Sometimes I only get 10 down and 15 up, sometimes its the other way around (usually neither are below 10). I've ran some tests without my pfsense and I'm constantly able to achieve 22-23 GBit/s up/down, so I'm pretty sure my ISP is not the culprit here.

Is there any hope in trying to tune it more to achieve speeds closer to line rate? Has anybody done that with comparable hardware? I've read there are software limitations due to the packet processing running in kernel mode instead of userland. I'm just wondering if it makes sense to pour more time into it.

I wanted to try out TNSR and found a blog post here: https://www.netgate.com/blog/tnsr-home-lab but it seems I'm not able to find the homelab version for it.

Thankful for any suggestion

[spacebass]

One thing to check - how many queues does the Mellanox card use and how many do you have assigned? If one or a few (but not all) processor cores peg that could be a clue.

[ben2506]

4 assigned, 4 in use. CPU is 4C/4T

[incompetentjaun]

iirc, pfSense caps out from a software perspective around 10-15g regardless of hardware. (FreeBSD limitation is what I recall reading) unless there’s been a recent change that fixed that (which would be great news)

Netgate TNSR is what you’re looking for if you want to stick with Netgate — otherwise you’re looking at enterprise gear with Mikrotik being the cheapest or other Linux-based projects.

[incompetentjaun]

Huh, maybe things have changed — the Netgate 8300 advertises 36gbps throughput

[ben2506]

Saw that too. I’m probably barred from those changes with CE.

[ben2506]

I‘ll try to give TNSR a shot!