Interpretation of NIO's Buffer

A buffer is essentially a block of memory that data can be written to and then read from. This piece of memory is packaged as a NIO Buffer object, and a set of methods are provided for easy access to this piece of memory. The buffer is actually a container object, more directly, it is actually an array . In the NIO library, all data is processed by the buffer. When data is read, it is directly read into the buffer; when data is written, it is also written into the buffer; whenever data in NIO is accessed, it is put into the buffer. In a stream-oriented I/O system, all data is directly written or read directly into the Stream object.

In NIO, all buffer types inherit from the abstract class Buffer, and the most commonly used one is ByteBuffer. For the basic types in Java, there is basically a specific Buffer type corresponding to them. The inheritance relationship between them is shown in the figure below Show:

Basic usage of Buffer

Steps for usage

1. Use Buffer to read and write data, generally follow the following four steps:

(1) Write data to Buffer

(2) Call the flip() method

(3) Read data from Buffer

(4) Call the clear() method or the compact() method

When writing data to the buffer, the buffer will record how much data is written. Once the data is to be read, the Buffer needs to be switched from write mode to read mode through the flip() method. In read mode, all data previously written to the buffer can be read. Once all the data has been read, the buffer needs to be cleared so it can be written again. There are two ways to clear the buffer: call the clear() or compact() method. The clear() method will clear the entire buffer. The compact() method will only clear the data that has already been read. Any unread data is moved to the beginning of the buffer, and newly written data is placed after the unread data in the buffer.

Example using Buffer

  @Test
    public void testConect2() throws IOException {
        RandomAccessFile aFile = new RandomAccessFile("d:\\atguigu/01.txt", "rw");
        FileChannel inChannel = aFile.getChannel();

        //create buffer with capacity of 48 bytes
        ByteBuffer buf = ByteBuffer.allocate(48);
        int bytesRead = inChannel.read(buf); //read into buffer. 
        while (bytesRead != -1) {
            buf.flip(); //make buffer ready for read
            while(buf.hasRemaining()){
                System.out.print((char) buf.get()); // read 1 byte at a time
            }buf.clear(); //make buffer ready for writing
            bytesRead = inChannel.read(buf);
        }
        aFile.close();
    }

Example using IntBuffer

     @Test
    public void testConect3() throws IOException {
        // 分配新的 int 缓冲区，参数为缓冲区容量
        // 新缓冲区的当前位置将为零，其界限(限制位置)将为其容量。
        // 它将具有一个底层实现数组，其数组偏移量将为零。
        IntBuffer buffer = IntBuffer.allocate(8);
        for (int i = 0; i < buffer.capacity(); ++i) {
            int j = 2 * (i + 1);
        // 将给定整数写入此缓冲区的当前位置，当前位置递增
            buffer.put(j);
        }
        // 重设此缓冲区，将限制设置为当前位置，然后将当前位置设置为 0
        buffer.flip();
        // 查看在当前位置和限制位置之间是否有元素
        while (buffer.hasRemaining()) {
        // 读取此缓冲区当前位置的整数，然后当前位置递增
            int j = buffer.get();
            System.out.print(j + " ");
        }
    }

Buffer capacity, position and limit

In order to understand how a Buffer works, you need to be familiar with its three properties:

- Capacity

- Position

- limit

The meaning of position and limit depends on whether the Buffer is in read mode or write mode. No matter what mode the Buffer is in, the meaning of capacity is always the same.

Here is an explanation about capacity, position and limit in read and write mode:

capacity

As a memory block, Buffer has a fixed size value, also called "capacity". You can only write capacity bytes, long, char and other types into it. Once the Buffer is full, it needs to be cleared (by reading or clearing the data) before continuing to write data into it.

position

1) When writing data into the Buffer, position indicates the current position of the written data, and the initial value of position is 0. When a byte, long, etc. data is written to the Buffer, the position will move down to the next Buffer unit where the data can be inserted. The position can be up to capacity - 1 (because the initial value of position is 0).

2) When reading data into the Buffer, position indicates the current position of the read data. For example, when position=2, it means that 3 bytes have been read, or the reading starts from the third byte. When switching to the read mode through ByteBuffer.flip(), the position will be reset to 0. When the Buffer reads data from the position, the position will move down to the next data Buffer unit that can be read.

limit

1) When writing data, limit indicates how much data can be written to the Buffer at most. In write mode, limit is equal to the capacity of Buffer.

2) When reading data, limit indicates how much readable data (not null data) is in the Buffer, so all data written before can be read (limit is set to the amount of written data, this value is in write mode position).

Buffer type

- ByteBuffer

- MappedByteBuffer

- CharBuffer

- DoubleBuffer

- FloatBuffer

- IntBuffer

- LongBuffer

- ShortBuffer

These Buffer types represent different data types. In other words, the bytes in the buffer can be manipulated by char, short, int, long, float or double types.

Buffer allocation and reading and writing data

Buffer allocation

To get a Buffer object, you must first allocate it. Each Buffer class has an allocate method. Below is an example of a ByteBuffer that allocates a capacity of 48 bytes.

ByteBuffer buf = ByteBuffer.allocate(48);

This is allocating a CharBuffer that can store 1024 characters:

CharBuffer buf = CharBuffer.allocate(1024);

Write data to Buffer

There are two ways to write data to Buffer:

(1) Write from Channel to Buffer.

(2) Write to Buffer through the put() method of Buffer.

Example of writing from Channel to Buffer:

int bytesRead = inChannel.read(buf); //read into buffer.

An example of writing a Buffer through the put method:

buf.put(127);

There are many versions of the put method, allowing you to write data into the Buffer in different ways. For example, write to a specified location, or write a byte array to Buffer

flip() method

The flip method switches the Buffer from write mode to read mode. Calling the flip() method will set the position back to 0 and set the limit to the value of the previous position. In other words, position is now used to mark the reading position, and limit indicates how many bytes, chars, etc. have been written before (how many bytes, chars, etc. can be read now).

Read data from Buffer

There are two ways to read data from Buffer:

(1) Read data from Buffer to Channel.

(2) Use the get() method to read data from the Buffer.

An example of reading data from Buffer to Channel:

//read from buffer into channel. 
int bytesWritten = inChannel.write(buf);

An example of using the get() method to read data from Buffer

byte aByte = buf.get();

There are many versions of the get method, allowing you to read data from the Buffer in different ways. For example, read from the specified
position, or read data from Buffer to byte array.

Several methods of Buffer

rewind() method

Buffer.rewind() sets the position back to 0, so you can reread all the data in the Buffer. limit remains unchanged, and still indicates how many elements (byte, char, etc.) can be read from the Buffer.

clear() and compact() methods

Once the data in the Buffer is read, the Buffer needs to be ready to be written again. This can be done with the clear() or compact() methods.

If the clear() method is called, position will be set back to 0 and limit will be set to the value of capacity. In other words, the Buffer is emptied. The data in the Buffer is not cleared, but these marks tell us where to start writing data into the Buffer.

If there is some unread data in the Buffer, call the clear() method, and the data will be "forgotten", which means that there is no longer any mark that will tell you which data has been read and which has not.

If there are still unread data in the Buffer, and these data are needed later, but you want to write some data first, then use the compact() method.

The compact() method copies all unread data to the beginning of the Buffer. Then set the position to just after the last unread element. The limit attribute is still set to capacity like the clear() method. Now the Buffer is ready to write data, but will not overwrite unread data.

mark() and reset() methods

A specific position in the Buffer can be marked by calling the Buffer.mark() method. It can be restored to this position later by calling the Buffer.reset() method. For example:

buffer.mark();
//call buffer.get() a couple of times, e.g. during parsing. 
buffer.reset(); //set position back to mark.

buffer operation

buffer fragmentation

In NIO, in addition to allocating or wrapping a buffer object, you can also create a sub-buffer based on the existing buffer object, that is, cut out a slice on the existing buffer as a new buffer, but The existing buffer and the created sub-buffer share data at the underlying array level, that is to say, the sub-buffer is equivalent to a view window of the existing buffer. Call the slice() method to create a subbuffer.

    @Test
    public void testConect3() throws IOException {
        ByteBuffer buffer = ByteBuffer.allocate(10);
        // 缓冲区中的数据 0-9
        for (int i = 0; i < buffer.capacity(); ++i) {
            buffer.put((byte) i);
        }
        // 创建子缓冲区
        buffer.position(3);
        buffer.limit(7);
        ByteBuffer slice = buffer.slice();
        // 改变子缓冲区的内容
        for (int i = 0; i < slice.capacity(); ++i) {
            byte b = slice.get(i);
            b *= 10;
            slice.put(i, b);
        }
        buffer.position(0);
        buffer.limit(buffer.capacity());
        while (buffer.remaining() > 0) {
            System.out.print(buffer.get()+" ");
        }
    }

read-only buffer

Read-only buffers are very simple, you can read them, but you cannot write to them . You can convert any regular buffer into a read-only buffer by calling the buffer's asReadOnlyBuffer() method. This method returns a buffer that is exactly the same as the original buffer and shares data with the original buffer, except that it is only read. If the content of the original buffer changes, the content of the read-only buffer also changes:

    @Test
    public void testConect4() throws IOException {
        ByteBuffer buffer = ByteBuffer.allocate(10);
        // 缓冲区中的数据 0-9
        for (int i = 0; i < buffer.capacity(); ++i) {
            buffer.put((byte) i);
        }

        // 创建只读缓冲区
        ByteBuffer readonly = buffer.asReadOnlyBuffer();
        // 改变原缓冲区的内容
        for (int i = 0; i < buffer.capacity(); ++i) {

            byte b = buffer.get(i);
            b *= 10;
            buffer.put(i, b);
        }
        readonly.position(0);
        readonly.limit(buffer.capacity());
        // 只读缓冲区的内容也随之改变
        while (readonly.remaining() > 0) {
            System.out.println(readonly.get());
        }
    }

If you try to modify the content of the read-only buffer, a ReadOnlyBufferException will be reported. Read-only buffers are useful for protecting data. When passing a buffer to a method of an object, there is no way of knowing whether the method will modify the data in the buffer. Creating a read-only buffer ensures that the buffer will not be modified. You can only convert regular buffers to read-only buffers, not convert read-only buffers to writable buffers.

direct buffer

A direct buffer is a buffer that allocates memory in a special way to speed up I/O. The description in the JDK documentation is: Given a direct byte buffer, the Java virtual machine will do its best to directly allocate it Perform native I/O operations. That is, it tries to avoid copying the contents of the buffer to or from an intermediate buffer before (or after) each call to the underlying operating system's native I/O operation . To allocate a direct buffer, you need to call the allocateDirect() method instead of the allocate() method, which is no different from a normal buffer.

Copy file example:

    @Test
    public void testConect5() throws IOException {
        String infile = "d:\\atguigu\\01.txt";
        FileInputStream fin = new FileInputStream(infile);
        FileChannel fcin = fin.getChannel();
        String outfile = String.format("d:\\atguigu\\02.txt");
        FileOutputStream fout = new FileOutputStream(outfile);
        FileChannel fcout = fout.getChannel();
        // 使用 allocateDirect，而不是 allocate
        ByteBuffer buffer = ByteBuffer.allocateDirect(1024);
        while (true) {
            buffer.clear();
            int r = fcin.read(buffer);
            if (r == -1) {
                break;
            }
            buffer.flip();
            fcout.write(buffer);
        }
    }

memory-mapped file I/O

Memory-mapped file I/O is a method of reading and writing file data that can be much faster than regular stream-based or channel-based I/O. Memory-mapped file I/O is done by making the data in the file appear as the contents of a memory array, which at first sounds like nothing more than reading the entire file into memory, but it is not. In general, only the portion of the file that is actually read or written is mapped into memory.

static private final int start = 0;
static private final int size = 1024;
static public void main(String args[]) throws Exception {
    RandomAccessFile raf = new RandomAccessFile("d:\\atguigu\\01.txt", "rw");
    FileChannel fc = raf.getChannel();
    MappedByteBuffer mbb = fc.map(FileChannel.MapMode.READ_WRITE, 
    start, size);
    mbb.put(0, (byte) 97);
    mbb.put(1023, (byte) 122);raf.close();
}

ByteBuffer size allocation

Each channel needs to record messages that may be split, because ByteBuffer cannot be shared by multiple channels, so it is necessary to maintain an independent ByteBuffer for each channel
ByteBuffer cannot be too large. For example, if a ByteBuffer is 1Mb, it needs 1Tb of memory to support millions of connections, so it is necessary to design a ByteBuffer with variable size.
- One way of thinking is to first allocate a smaller buffer, such as 4k. If the data is found to be insufficient, then allocate an 8k buffer and copy the contents of the 4k buffer to the 8k buffer. The advantage is that the message is continuous and easy to process, and the disadvantage is that data copying consumes performance. Refer to Implementing Java Resizable Array Sometimes you want to keep data in a single, consecutive array for fast and easy access, but need the array to be resizable, or at least expandable. This tutorial shows you how to implement a resizable array in Java. http : //tutorials.jenkov.com/java-performance/resizable-array.html
- Another way of thinking is to use multiple arrays to form a buffer. One array is not enough, and the extra content is written into a new array. The difference from the previous one is that the message storage is discontinuous and the analysis is complicated. The advantage is that it avoids the performance loss caused by copying.