How fast is it to copy a file through zero-copy technology?

concept

As a file copy technology, zero copy refers to a type of technology rather than a technical framework. It is usually implemented in two ways. One is DMA (direct memory access), which is translated as direct memory access. A hardware-level technology. Its principle is that no CPU intervention is required when copying files. The files can be copied directly from the memory to another hardware, such as a disk or a network card. Another technology is MMAP ( Memory Mapping) is a technology that maps files or other resources to the address space of a process, so that the process can access these resources like ordinary memory. It is a software-level technology that requires a CPU compared to traditional file copying. Without intervention, the file data needs to be copied several times in the kernel space and user space. Zero-copy technology has obvious advantages in file copying.

experiment

Let's conduct experiments through code. We also copy a zip file (768M) downloaded from the idea official website and compare the efficiency of the three copies:

    private File file = new File("D:\\java开发\\ideaIU-2019.3.5.win.zip");
    private String targetPath = "D:\\java_code\\filecopy\\";

1. Traditional copy method: average time is about 9 seconds

    @Test
    public void testInputstream() throws Exception {
        FileInputStream fileInputStream = new FileInputStream(file);
        File targetFile = new File(targetPath + "inputstream.zip");
        FileOutputStream fileOutputStream = new FileOutputStream(targetFile);
        byte[] bytes = new byte[1024];
        StopWatch stopWatch = new StopWatch();
        stopWatch.start();
        while (fileInputStream.read(bytes) != -1) {
            fileOutputStream.write(bytes);
        }
        stopWatch.stop();
        System.err.println(stopWatch.toString());
    }

 2. Use BufferedOutputStream to buffer the output stream copy method: the average time is about 2 seconds 4. BufferedOutputStream will use an 8kb array as the buffer by default, so that you don’t have to flush the disk through a system call every time you write a byte. You can Reducing io is naturally faster than the first method, but I have tried adjusting the size of this buffer, and there is not much difference in time consumption.

    @Test
    public void testBuffedInputstream() throws Exception {
        FileInputStream fileInputStream = new FileInputStream(file);
        File targetFile = new File(targetPath + "bufferedinputstream.zip");
        FileOutputStream fileOutputStream = new FileOutputStream(targetFile);
        BufferedOutputStream bufferedOutputStream = new BufferedOutputStream(fileOutputStream, 8192 * 4);
        byte[] bytes = new byte[1024];
        StopWatch stopWatch = new StopWatch();
        stopWatch.start();
        while (fileInputStream.read(bytes) != -1) {
            bufferedOutputStream.write(bytes);
        }
        bufferedOutputStream.flush();
        stopWatch.stop();
        System.err.println(stopWatch.toString());
    }

3. MMAP method: average time is about 2 seconds

@Test
    public void testMmap() throws Exception {
        FileChannel sourceChannel = new RandomAccessFile(file, "r").getChannel();
        File targetFile = new File(targetPath + "mmap.zip");
        FileChannel targetChannel = new RandomAccessFile(targetFile, "rw").getChannel();
        FileOutputStream fileOutputStream = new FileOutputStream(targetFile);
        long size = sourceChannel.size();
        // 创建源文件和目标文件的 MappedByteBuffer
        MappedByteBuffer sourceBuffer = sourceChannel.map(FileChannel.MapMode.READ_ONLY, 0, size);
        MappedByteBuffer targetBuffer = targetChannel.map(FileChannel.MapMode.READ_WRITE, 0, size);
        // 使用 MappedByteBuffer 拷贝文件
        StopWatch stopWatch = new StopWatch();
        stopWatch.start();
        for (long i = 0; i < size; i++) {
            byte b = sourceBuffer.get();
            targetBuffer.put(b);
        }
        stopWatch.stop();
        System.err.println(stopWatch.toString());
    }

 4. DMA method (I am not sure whether this is DMA, because it is a hardware-level technology and cannot be directly verified. It depends on the operating system and hardware): the average time is about 1 second.

    @Test
    public void testDma() throws Exception {
        File targetFile = new File(targetPath + "dma.zip");
        FileChannel sourceChannel = new FileInputStream(file).getChannel();
        FileChannel targetChannel = new FileOutputStream(targetFile).getChannel();
        StopWatch stopWatch = new StopWatch();
        stopWatch.start();
        // 使用transferFrom或transferTo方法拷贝文件
//        targetChannel.transferFrom(sourceChannel, 0, sourceChannel.size());
        // 或者
        sourceChannel.transferTo(0, sourceChannel.size(), targetChannel);
        stopWatch.stop();
        System.err.println(stopWatch.toString());
    }

 Summarize:

1. Zero-copy technology is obviously much faster than traditional file copying, and DMA implemented at the hardware level is faster than MMAP at the software level.

2. Applicable scenarios: those that do not require CPU intervention for calculations, such as mq message middleware. When a consumer consumes a message, the mq server only needs to read the message and send it to the consumer without performing calculations. The more famous kafka Just use DMA, and rocketmq uses MMAP