Zero-Copy for Linux

Reference:  https://segmentfault.com/a/1190000011989008

 

What zero copies are?

Wikipedia on the "zero copy" described it this way:

"Zero-copy" describes computer operations in which the CPU does not perform the task of copying data from one memory area to another. This is frequently used to save CPU cycles and memory bandwidth when transmitting a file over a network.

"Zero-copy" is the CPU does not execute the operation to copy data from a computer memory area to an area other tasks described. It is typically used to save CPU cycles and memory bandwidth when transmitting the file over the network. Briefly, it is a way of avoiding zero-copy copying data from one CPU to memory storage of an additional technology.

Why do we need zero-copy technology?

Usually we have such a demand: a file on the local disk sent to the other remote services over the network. In traditional I / O, the operating system we are by what happened a look at the chart:

1. Issuing read () system call, then the processor switches from user space to kernel space;

2. Requests data from the disk;

3. By the DMA read from the disk file space to the kernel buffer;

4. read () system call returns, the data is copied from the kernel buffer to the user-space buffer space, this time the processor switches from kernel space to user space;

5. Issue write () system call, and the data is copied from the buffer to the target user space socket buffer in the kernel space, this time the processor switches from user space to kernel space;

6. write () call returns;

7. By copying data from the DMA buffer in kernel space to the protocol engine (which is independent and asynchronous operation).

Overall: The traditional I / O operation will have a context switches 4 and 4 copies of data throughout the entire process.

Q: Some people might ask, why write () call will first return, did not he would return before the data transfer?
A: In fact return the call does not guarantee that data is transmitted, and even he is not guaranteed to start transmission, Ethernet driver simply means there is space in its transmission queue and we will have to accept the data to be transferred. It is likely to exclude many packets before us. Unless the driver or the hardware priority queue or ring otherwise data will be transmitted FIFO manner.

Learn the traditional I / O operations, we come to look at the whole process, we will notice that the second and third copy of the data is completely pointless, application caching only a bit of the data is intact and it is sent to the target socket buffer. And these two copies are required CPU full participation from the operating system point of view, if the CPU has been occupied with to perform this simple task, then this would be a waste of resources; if there are other relatively simple system components can do it this matter, so that the CPU can be freed to do other things, then the use of system resources will be more effective.

"Zero copy" It is through the elimination of redundant copies to improve performance. In the data transfer process, so that data is copied, and the CPU copies the data in the buffers between the kernel space kernel space and user space buffer buffers.

---

Zero-copy implementation mechanism

Linux provides a similar system calls mainly sendfile (), mmap (), and splice () (This article calls the system temporarily do discuss).

Zero copy by the sendfile () implemented

sendfile system call kernel version 2.1 is introduced, the purpose is to simplify the process of data transfer carried out between two local files over the network. sendfile system call is introduced, not only reduces the replication of data, also reduces the number of context switches. For better description, see below:

1. Issuing the sendfile () system call, then the processor switches from user space to kernel space;

2. Requests data from the disk;

3. By the DMA read from the disk file space to the kernel buffer;

4. The data from the kernel buffer space copied to the destination socket buffer;

5. Sendfile () returns, when the processor is switched from the kernel space to user space;

6. DMA by copying data from the socket buffer to a target protocol engine.

总结一下这种实现，整个过程产生了2次上下文切换和3次数据拷贝（其中2次DMA拷贝和1次CPU拷贝）。

该实现虽然减少了2次上下文切换，但仍然还有1次CPU拷贝。那这次拷贝是不是也可以省掉呢？答案是肯定的。但是需要底层操作系统的一些支持。那就是带有DMA收集功能的sendfile实现的零拷贝。

带有DMA收集功能的sendfile实现的零拷贝

从Linux2.4开始，操作系统底层提供了带有scatter/gather的DMA来从内核空间缓冲区中将数据读取到协议引擎中。这就意味着等待传输的数据不需要在连续存储器中，它可以分散在不同的内存位置。那这样一来，从文件中读出的数据就不必拷贝至目标socket的缓冲区中，只需要将缓冲区描述符添加到目标socket的缓冲区中，DMA收集操作会根据缓冲区描述符中的信息将内核空间缓冲区中的数据读取到协议引擎。这种方法不仅减少了上下文切换、还减少了由CPU参与的数据拷贝。为了更好的理解这种方法所涉及的操作，请看下图：

1. 发出sendfile()系统调用，处理器从用户空间切换至内核空间;

2. 通过DMA将数据copy至内核空间缓冲区;

3. 将数据在内核空间缓冲区的地址和偏移量拷贝至目标socket的缓冲区;

4. Sendfile()返回，处理器从内核空间切换至用户空间。

5. 带有scatter/gather 功能的DMA将数据直接从内核缓冲区读取到协议引擎，从而消除了最后一次CPU拷贝。

总结一下，这种方法产生了2次上下文切换和2次数据拷贝。

这时有人可能会问，如果我把数据从磁盘上读出来后，再编辑一下，再发送出去，以上所说的零拷贝岂不是不能实现？
对于该问题，Linux提供了mmap来实现。

通过mmap实现的零拷贝

mmap(内存映射):mmap操作提供了一种机制，让用户程序直接访问设备内存，这种机制，相比较在用户空间和内核空间互相拷贝数据，效率更高。

1. 发出mmap()系统调用，处理器从用户空间切换至内核空间。

2. 向磁盘请求数据;

3. Through the DMA data is copied from the disk space to the kernel buffer;

4. mmap () call returns, this time the user programs and operating systems to share the buffer, then the data need not be copied from the kernel buffer to the user buffer, the processor is switched from the kernel space to user space;

5. User logic processing;

6. Issue write () system call, the kernel space data is copied from the buffer to the target socket buffer, then the processor switches from user space to kernel space;

7. write () call returns, the processor is switched from the kernel space to user space;

8. By copying the data to the DMA protocol engine.

To summarize: This method will generate context switches 4 and 3 copies of the data.

So far, zero-copy technology has introduced over. This article mentioned zero-copy technology are needed to the underlying operating system support, while zero-copy technology has been constantly developed and perfected them, this article does not cover all zero-copy technology appeared on Linux.