It doesn't mention in the article why the kernel is so slow at processing network packets. I'm not a kernel programmer so this may be utterly wrong, but wouldn't it be possible to sacrifice some feature for speed by disabling it in the kernel code?
One thing to remember is that high-speed packet filtering is an unusual workflow and CloudFlare operates at a much greater scale than most of us see: most Linux devices are not connected to 10G, much less 100G, networks and they're usually doing more work than looking at a packet to decide whether to accept or reject it. The fact that the APIs and the kernel stack were designed many years before those kind of speeds were possible doesn't matter because most sites don't have that much traffic and most server applications will bottleneck at doing other work well before that point.
The example in the article found a single core handling 1.4M packets per second. If you're running a web-server shoveling data out to clients those packets are going to be close to the maximum size which, if I haven't screwed up the math, looks something like this:
That's not to say that there isn't still plenty of room to improve and, as lukego noted, there's a lot of work in progress (see e.g. https://lwn.net/Articles/615238/ on work to batch operations to avoid paying some of the processing costs for every packet) but for the average server you'd find bottlenecks on something like a database, application logic, request handling, client network capacity, etc. before the network stack overhead is your greatest challenge. The people who encounter this tend to be CDN vendors like CloudFlare and security people who need to filter, analyze, or generate traffic on levels which are at least the the scale of a large company (e.g. https://github.com/robertdavidgraham/masscan).
People are also spending hundreds of billions of dollars on equipment like routers ever year. These could be Linux boxes if the kernel had sensible performance.
I am kind of amazed that the Linux kernel did not become the dominant data-plane for the networking industry ahead of proprietary implementations from Cisco, Juniper, etc.
Hopefully Snabb Switch will have better luck there... ;-)
Most switch/router dataplane processing is done in hardware, with control by custom drivers in the control-plane OS.
Cisco IOS is variously hosted on a proprietary real-time OS, QNX or Linux.
Juniper JunOS is FreeBSD-based.
Arista AOS is Linux-based.
With the advent of SDN, the packet-rate limitations of a general-purpose OS lead to things like DPDK and Snabb, but they both run within a Linux host environment (DPDK can use FreeBSD as well; not sure re: Snabb).
When 100M ethernet was introduced in the 90s, PCs had Pentiums, at < 100 M instructions per second. Both x86 CPUs and Ethernet have gotten ~1000x faster in the same time - cf 100G Ethernet and processors with 30x frequency, ~2x more IPC, and 20x more cores.
You haven't accounted for memory bandwidth and latency. And counting increased core count means you can't actually assume the same old single threaded algorithm will scale linearly.
Those are real-world speed bumps that make sense when you want to do stateful processing w/ large working sets, but not really for this "just get packets through the stack" toy benchmark that fits in cache and the workload is parallel-friendly.
And also the article is talking about handling just 10G, not 100G!
> One thing to remember is that high-speed packet filtering is an unusual workflow and CloudFlare operates at a much greater scale than most of us see: most Linux devices are not connected to 10G
You're about to see a whole bunch more 10G in the coming 5 years.
> Sorry if I sound mean, but this is just a long apologist post about how things are just so hard. Really? Why? Why can't Linux match BSD's performance?
Mostly I just wish you'd read it again: you appear to have missed the part where I said that this is a real problem which needed working on.
The point I was making is that it's not a problem for most Linux users. Linux includes millions of devices attached to sub-100Mb networks but even if you want to look solely at things in modern data centers ask yourself how many of them are running network-limited applications or are providing services to internet users over an uplink which is actually fast enough to stress modern hardware. For all but the most demanding users the Linux vs. BSD decision will be made on other factors.
Those millions of sub-100Mb network devices aren't pushing Linux forward though. Sure, they're using Linux, but they're not the ones pushing it forward. It's colo and datacenter use cases that are. A basic MySQL OLTP load with SSD's and a modicum of compute power will easily saturate a 10gig NIC. It does start to die shortly after that due to network/kernel stuff. I'd really like to get a lot more out of my existing hardware if i could.
You mean “pushing it forward in this particular direction”. Again, it's great to have people working on this and better if the companies using Linux support developers working on the problem but it's only a deal-breaker for a much smaller number of people.
Just to use an example which you see a lot today: how many of the developers jumping on Docker care that much about network performance at this level? I would argue that continued development of the container system has done more to boost Linux usage than low-level networking performance, even though both are entirely legitimate and worthwhile concerns worth developer sponsorship. (ZFS has pulled in the opposite direction for people who care about storage)
From the other direction, imagine if *BSD had gotten serious about package management by the mid-to-late 90s when it was obvious how much better the experience was on Debian so that a generation of developers wasn't trained to favor Linux to avoid getting sucked into dependency management. That doesn't have anything to do with the kernel but it mattered more for many, many people. This would have been really interesting if kfreebsd had hit critical mass and made the cost of switching that much lower.
Well the BSDs have issues too. FreeBSD/Netflix has been doing in kernel SSL to try to fix some issues, and there is Netmap available for packet processing in userspace. So they are running into similar issues as Linux.
Doesn't look like anyone has really answered your question yet.
There are three main reasons the kernel is slow for networking: per-packet dynamic memory allocation, lots of memory copying, and system call overheads.
The first two can be improved by modifying the kernel, and I think people are attempting to do this. The system call overheads arise naturally from having the networking code in the kernel. Basically every time you perform a system call the kernel has to save the userspace context, do the system call, and restore the context. This takes time and is bad for cache locality.
But as others noted, for most people who aren't cloudflare this doesn't really matter.
>But as others noted, for most people who aren't cloudflare this doesn't really matter.
Aren't most web applications I/O bound? The Arrakis team sped up Memcached and Haproxy quite a lot by bypassing the kernel. It seems like there could be a large market for these techniques as they become easier to use.
Hmmm, that's a good point. I was thinking more of a typical webapp, but you're probably right that there are certain classes of applications (e.g. caching) that are I/O bound under load.
Reason #4: Another problem is that the network kernel was never designed to do internet on the mass scale desired today. Companies like whatsapp devoted lots of time to getting e.g. 2M concurrent TCP connections (considered good) running on a single box, mainly because of the greedy overhead and design of the legacy network kernel. Whereas, in theory it should be possible to have 10M or more concurrent TCP connections on modern average hardware. So from this POV then the legacy network kernel is the bloated memory greedy mess that Java is to software development. See http://c10m.robertgraham.com/p/manifesto.html
Note they cite higher numbers as the current status than the Cloudfare article. In the LWN article it says "The kernel, today, can only forward something between 1M and 2M packets per core every second"
It isn't as slow as they claim, just like was discussed in HN comments to the predecessor article ("How to receive a million packets per second"). Still they repeat the general claim of "Vanilla Linux can do only about 1M pps". Makes for better headlines I guess.
tldr: cloudflare like to reimplement things and claim it solves the world problems. Thats cool, thats how open source work. Fork, copy, reimplement, try new stuff.
that-said...slightly-longer: for what they do there are alternatives like ipset
for other things its not as clear-cut, hence things like PF_RING. its not that great thought, you're sacrificing all features for fast sniffing.
technically a good zero-copy implementation of packet mmap w/ a userspace ring would achieve +- the same thing, too.
