C1000k New Ideas: User Mode TCP/IP Protocol Stack
It is not news that the current server supports millions of concurrent TCP connections ( Yu Feng's speech in 2010, ideawu 's iComet open source project , WhatsApp achieved 2.5M ). A common practice for implementing C1000k is to adjust kernel parameters, increase the number of files, and reduce memory consumption per connection ( refer to ideawu's blog ).
At this year's BSDCan2014 conference, Patrick Kelsey introduced the porting of FreeBSD 9.x's TCP/IP stack to user mode ( slides , github.com/pkelsey/libuinet ) and used it in the WANProxy project. Running the TCP/IP protocol stack in user mode means that concurrent TCP connections no longer occupy system files, but only memory, which solves a major bottleneck of C1000k. The kernel only needs to provide an interface for sending and receiving network packets (such as netmap ).
The kernel's network protocol stack emphasizes versatility, mainly for throughput optimization (the performance indicator is usually MB/s or packets per second), and by the way, a large number of concurrent connections are taken into account. In order to support the C1000k, it is necessary to adjust the kernel parameters so that each connection occupies less resources, which is contrary to the original intention of the kernel code.
The user mode protocol stack pierces this layer of window paper, and the function of the protocol stack can be tailored according to the characteristics of the application. The optimization is also more direct, instead of adjusting the black box parameter combination, it is directly profiling, and the code of the application and protocol stack is modified according to the results.
The throughput of the user mode protocol stack is not as good as that of the kernel, but it should not be a problem for C1000k applications (such as comet).
muduo's C1000k experiment
I used muduo to do a C1000k experiment, using the traditional scheme, not using libuinet. 500,000 TCP connections were created on an old Dell WS490 workstation with 16GB of RAM, providing echo services . The available memory of the system is reduced by 5286MiB, that is, 10.8KiB per connection (the service process occupies 1421MiB of memory, that is, 2.9KiB per connection, and the remaining 8KiB is the overhead of the kernel protocol stack). The client is an i5-2500 with 8GB memory, and the memory consumption is also more than 5GB, so this experiment only tried C500k. The client is bound to 10 IPs, each IP sends out 50,000 TCP connections, runs the pingpong protocol, and each connection sends and receives 64-byte messages in turn. The measured QPS is about 11k, and the server's CPU usage is about 60% (single thread). The profile shows that the main CPU overhead is in the kernel, and I'm mostly happy with the results.
Revive 4.4BSD-Lite2's network protocol stack
Inspired by libuinet, I also ported the 4.4BSD-Lite2 network protocol stack to Linux user mode ( github.com/chenshuo/4.4BSD-Lite2 ), which is convenient for readers of "TCP/IP Detailed Explanation Volume 2" to track and debug its code. Below is a screenshot of the Eclipse CDT step trace.
You can also use various ready-made tools to analyze the calling relationship of functions:
In "Talking about Network Programming Learning Experience", I said in this book that "the code can only be read, not run on the computer, and cannot be changed to test" is no longer true.
I used TAP/TUN as the external interface of the protocol stack I wrote in "Several Thinking Questions and Experiments About TCP Concurrent Connections". I can do the same for 4.4BSD-Lite2, so that the TCP/IP protocol stack 20 years ago can be Machine communication now. In addition to communicating with this machine, it can also forward through NAT, so that 4.4BSD-Lite2 can connect to the current Internet. (sudo iptables -t nat -A PREROUTING -p tcp --dport 2009 -i eth0 -j DNAT --to 192.168.0.2:2009 )