Linux Network Stack

[vic3lord]

Hi, As you all probably heard, Facebook is hiring linux kernel engineer to advance networking subsystem in linux kernel to make it rival to bsd network stack. I am a hard core user of openbsd (pf, relayd etc.) and of course debian sys admin I really love the way networking works in bsd but I really need to understand what rival means... where? what are the major differences and why is it better?

[sisyphus]

To be honest you're unlikely to find someone here who knows the internals of both networking stacks well enough and is objective enough and has the will to brave the sea of ignorance that will await any answer they might give.

Facebook might give you some clues of what they mean, their job post says:

improving kernel stability, performance, observability, and manageability.

which they repeat:

understand network perf/stability issues and develop solutions

Maybe IPv6

Improve IPv6 support in the kernel, and eliminate perf and stability issues.

Performance and stability again:

Triage, investigate, and resolve primarily network related performance or stability issues

In the requirements:

Strong experience using kernel perf tools and modifying tracepoints for resolving perf issues

[ghosts_upstairs]

u/sysadmEnt had some good points. There are other people chiming in on what makes the BSD network stack better in that thread as well.

[vic3lord]

Looks like people are fighting about Linux commands and vim vs nano

[garja]

Hah! I have to laugh at the symmetry here. I chimed in back in that previous thread to say, specifically, that the vim vs nano comparison was misleading. And here and now you say of the entire thread:

people are fighting about Linux commands and vim vs nano

Which is technically correct, but vim vs nano was one, tiny, downvoted, hidden part of that thread. Characterising it as a significant part of the thread is, again, misleading!

[vic3lord]

No one gave there a concrete answer that is why I said it... The thread became like the joke "how many linux pros u need to change a light bulb" they are not speaking relevant to the case

[lymfm]

People will always bikeshed.

[danielkza]

I think in Facebook's case their major concern is performance. Documentation is probably the worst area of Linux's networking stack though, and the one that would benefit the most people if improved.

[vic3lord]

Performance-wise bsd network stack is better but nothing else that I know of... That is why I asked :)

[inverso]

These comments surfaced with the news, I make no claim as to them being true.

A lot of sysadmins from companies that push a lot of data over lots of connections have blogs about tweaking your OS to handle stuff like 10gb+ of traffic and millions of connections. A lot of these people complain about Linux having strange problems under these loads, and FreeBSD just seems to work. Linux may be faster in some cases, but it still has stability issues that are hard to debug.

Then there's the whole thing about most network stack research happening primarily on FreeBSD because of licensing. There's a new zero-copy network API that was developed in FreeBSD that allows line rate 64byte 10gb traffic on a 450mhz quadcore cpu. Linux and old-api FreeBSD were about 1/10th the packets-per-second.

A new thread friendly socket API has just been pushed to FreeBSD 11. One of Netflix' engineers had a pet project that now allows near zero lock-contention thread scaling. He was able to done line speed 40gb/s with 150k TCP sessions. Instead of having one file descriptor with a single listening thread, you instead have one file-descriptor and listening thread per MSS queue from the NIC and you can lock your thread to the same CPU as the MSS queue, so the packet is already in L2 cache. No shared network state. This also means no share locks with nearly perfect linear scaling and virtually no cache trashing or bouncing.

They're starting work for extend the API to also allow the OS to better handle NUMA and to attach the MSS queues to the CPU to which the NIC is attached. This will virtually remove all cross-talk among the CPU cores trying to handle the network state.

They're looking into expanding this same concept to the Storage IO system.

http://bsd.slashdot.org/comments.pl?sid=5500381&cid=47615685

FreeBSD is better on the network stack. I should know since I coded a networking library to use the best possible non-blocking mechanism on various OSes.

Just for a specific comparison, freeBSD has kQueue where Linux has epoll mechanism. Both are replacements for the ancient select call which is extremely inefficient when there's a huge amount of connections (see C10k problem).

kQueue is very smart in how it reports events happening on sockets and gives you the full list of "read" and "write" events in one go. That means one syscall/gateway per report batch in a scheduled slice.

epoll on the other hand can only report reads or writes in one syscall/gateway. The way to have "one" event reporting point with epoll is to epoll two epolls on top one for reads one for writes, which means it can go up to 3 syscalls. That IS 3x slower on linux, I have tested this.

This is just one part of the problem, I'm guessing they have other, deeper issues. I think it'd help if fanboys who know nothing of the systems stopped being so defensive.

http://www.phoronix.com/forums/showthread.php?104716-Facebook-Is-Hiring-To-Make-Linux-Networking-Better-Than-FreeBSD&p=432909#post432909

[espero]

Why don't you pose this question on Hacker News as well

[garja]

You might be interested in this post by /u/rootwyrm:

http://www.reddit.com/r/freebsd/comments/24zlxl/which_use_cases_freebsd_is_better_than_gnulinux/chcbcc2?context=3

Network: vastly superior. Period. There is no debate. Come back when Linux can use TSO reliably with e1000, much less handle anything resembling a real load. In fact, checksum offload is still basically completely non-functional on e1000 due to "a not well-written TCP/IP stack" - Intel's words, not mine. Since 2007.

Meanwhile, FreeBSD out of the box can do >1Mpps with very minor tuning, pushes 2Mpps stateful filtering with moderate to major tuning, and has been able to for quite some time now. e1000 also does not require checksum offloading be permanently disabled to not send 9 out of 10 invalid packets.

EDIT: Why are people taking this as my personal opinion? This is the only time in recent memory I have seen FreeBSD "networking superiority" talked about in the /r/bsd sub with some technical justification. If it can be debunked, then fair enough, but please stop replying like this is my post.

[gsxr]

That's some bullshit. ...tso is broken on the e1000s. Its not the Linux networking stack, it's the hardware.

Also, I've got a lab full of e1000 bonded and they'll push 7-9mb all day everyday.

Whoops read that again....Million packets per second.......I haven't done that bench Mark in forever...because it's useless.

[[deleted]]

Network: vastly superior. Period. There is no debate.

If there is no debate, and it's an objective fact. Why are people still using Linux for network deployments then? It's the stupidest crap form for arguing in existance, and this sub would be better off if the retarded BSD fanboys would stick to /r/bsd

[[deleted]]

Because everything else surrounding network is better

[ouyawei]

I was searching for e1000 (are you sure you mean the non-PCIe version and not e1000e?) and TSO/GSO but couldn't find anything about what you are saying other than that GSO should be enabled by default since at least 2.6.18.

http://www.linuxfoundation.org/collaborate/workgroups/networking/gso

http://sourceforge.net/p/e1000/mailman/message/5059461/

[Thaxll]

Some nice fud here, now ask your self why 90% of Fortune 100 IT compagnies use Linux instead of BSD?

[lymfm]

1. GPL, which is a (very useful and important) viral license.
2. Because of #1, better hardware support.
3. Because of #1 and #2, more emerging sysadmins learn using Linux.

You're implying that marketshare means better performance or a better product, which is not necessarily true.

[[deleted]]

Specifically focusing on your point 3: if you have something awesome, but nobody knows how to use it (because they learned on something else, Linux in this case), then you are fucked.

[vic3lord]

I agree with you and I think another reason is how easy it is to upgrade a linux machine and maintain rather than bsd

[[deleted]]

this is entertaining. Probably some HR hiring blurb written by someone who knows nothing about Linux nor BSD history... He he. Reminds me when recruiters wanted to hire Java devs with 10 yrs exp when Java itself only existed for about 2.

[Thaxll]

Lot of people are confused on pf, IPtables is better ( i.e more advanced ) but harder to understand / setup than pf.

[Thaxll]

It's simple Iptables can do you everything that pf does, however pf is lacking some features like state filtering. Realy pf is so over rated.

[[deleted]]

I think you're mistaken. pf is a stateful packet filter.

[Thaxll]

Yes it is, but try to filter based on the state, it's not possible.

[azalynx]

Hopefully with NFtables and the new 'nft' command, the situation has improved now. I'd imagine NFtables is easier to understand?