JAVA NIO study notes - 01

NIO Profile

1.Java NIO contains the following core components:
- Channels
- Buffers
- SelectorsSelectors
2. There are various types of Channel and Buffer in NIO. The following list is the main channel implementation class in Java NIO:
- FileChannel
  - For file read and write operations
- DatagramChannel
  - Used for data read and write operations in the UDP protocol
- SocketChannel
  - Used to read and write network data in the TCP protocol
- ServerSocketChannel
  - It can listen for TCP connections like a web server, and it creates a new SocketChannel object for each incoming connection.
3. NIO channels are similar to streams, but some differences are as follows:
- Channels can read and write at the same time, while streams can only read or write
- Channels can achieve asynchronous read and write data
- Channels can read data from the buffer and write data to the buffer

//第一步是获取通道。我们从 RandomAccessFile获取通道：
//FileInputStream fin = new FileInputStream( "readandshow.txt" );
RandomAccessFile aFile = new RandomAccessFile("data/nio-data.txt", "rw");
FileChannel inChannel = aFile.getChannel();
//下一步是创建缓冲区：
ByteBuffer buf = ByteBuffer.allocate(48);
//最后，需要将数据从通道读到缓冲区中
int bytesRead = inChannel.read(buf);
while (bytesRead != -1) {
  System.out.println("Read " + bytesRead);
  buf.flip();
  while(buf.hasRemaining()){
      System.out.print((char) buf.get());
  }
  buf.clear();
  bytesRead = inChannel.read(buf);
}
aFile.close();
//注意 buf.flip()的使用。首先数据读取到Buffer后，
//对其进行翻转(flip),然后再从中读取数据。

Buffer

Basic usage of Buffer
1. Using Buffer to read and write data generally follow the following four steps:
- 1. Write data to Buffer
- 2. Call the flip() method
- 3. Read data from Buffer
- 4. Call the clear() method or the compact() method
2. When writing data to the buffer, the buffer will record how much data is written. Once the data is to be read, the Buffer needs to be switched from the write mode to the read mode by the flip() method. In read mode, all data previously written to the buffer can be read.
3. Once all the data has been read, the buffer needs to be emptied so that it can be written again. There are two ways to clear the buffer: call the clear() or compact() method. The clear() method will clear the entire buffer. The compact() method will only clear data that has already been read. Any unread data is moved to the beginning of the buffer, and newly written data is placed after the unread data in the buffer.
4. Buffer's capacity, position and limit
- A buffer is essentially a block of memory to which data can be written and then read from it. This piece of memory is wrapped as a NIO Buffer object and provides a set of methods for easy access to this piece of memory.
- It has three properties to be familiar with:
  - capacity
    - As a memory block, Buffer has a fixed size value, also called "capacity". You can only write capacity bytes, long, char and other types into it. Once the Buffer is full, it needs to be emptied (by reading data or clearing data) to continue writing data to it.
  - position
    - When you write data to the Buffer, position represents the current position. The initial position value is 0. When a byte, long and other data are written to the Buffer, the position will move forward to the next Buffer unit where data can be inserted. position can be up to capacity – 1.
    - When reading data, it is also reading from a specific location. When switching the Buffer from write mode to read mode, the position is reset to 0. When reading data from the Buffer's position, the position moves forward to the next readable position.
  - limit
    - In write mode, the limit of the Buffer indicates how much data you can write to the Buffer at most. In write mode, limit is equal to the capacity of Buffer.
    - When switching Buffer to read mode, limit indicates how much data you can read at most. Therefore, when switching the buffer to read mode, limit will be set to the position value in write mode. In other words, you can read all the previously written data (the limit is set to the number of written data, which is the position in write mode)
5. Types of Buffers
- ByteBuffer
- MappedByteBuffer
- CharBuffer
- DoubleBuffer
- FloatBuffer
- IntBuffer
- LongBuffer
- ShortBuffer
Buffer types represent different data types. In other words, the bytes in the buffer can be manipulated through the char, short, int, long, float or double type.
6. Buffer allocation
- To obtain a Buffer object, you must first allocate it. Every Buffer class has an allocate method. Below is an example of allocating a ByteBuffer of 48 bytes capacity.
  - ByteBuffer buf = ByteBuffer.allocate(48);
- This is allocating a CharBuffer that can store 1024 characters:
  - CharBuffer buf = CharBuffer.allocate(1024);
7. Write data to the Buffer
- 1. There are two ways to write data to Buffer:
  - Write from Channel to Buffer.
    - int bytesRead = inChannel.read(buf); //read into buffer.
  - An example of writing a Buffer through the put method:
    - buf.put(127);
8. flip() method
- The flip method switches the Buffer from write mode to read mode. Calling the flip() method will set the position back to 0 and set the limit to the previous position value.
- In other words, position is now used to mark the read position, and limit indicates how many bytes, chars, etc. were written before - how many bytes, chars, etc. can be read now.
9. Read data from Buffer
- There are two ways to read data from Buffer:
  - 1. Read data from Buffer to Channel.
    - int bytesWritten = inChannel.write(buf);
  - 2. An example of using the get() method to read data from the Buffer
    - byte aByte = buf.get();
10.rewind() method
- Buffer.rewind() sets the position back to 0, so you can reread all the data in the Buffer. The limit remains the same, still indicating how many elements (byte, char, etc.) can be read from the Buffer.
11.clear() and compact() methods
- Once the data in the buffer has been read, the buffer needs to be ready to be written again. This can be done with the clear() or compact() methods.
- If the clear() method is called, position will be set back to 0 and limit will be set to the value of capacity. In other words, the Buffer is emptied. The data in the buffer is not cleared, but these flags tell us where to start writing data to the buffer.
- If there is some unread data in the Buffer, call the clear() method and the data will be "forgotten", meaning there will no longer be any markers that will tell you which data has been read and which has not.
- If there is still unread data in the Buffer, and the data is needed later, but you want to write some data first, use the compact() method.
- The compact() method copies all unread data to the beginning of the Buffer. Then set the position to just after the last unread element. The limit property is still set to capacity like the clear() method. The buffer is now ready to write data, but will not overwrite unread data.
12.mark() and reset() methods
- 1. By calling the Buffer.mark() method, a specific position in the Buffer can be marked. This position can then be restored by calling the Buffer.reset() method. E.g:

buffer.mark();

//call buffer.get() a couple of times, e.g. during parsing.

buffer.reset();  //set position back to mark.

2. equals() and compareTo() methods
- You can use the equals() and compareTo() methods for two Buffers.
- equals()
  - Two Buffers are equal when the following conditions are met:
    - have the same type (byte, char, int, etc.).
    - The number of bytes, chars, etc. remaining in the Buffer is equal.
    - All remaining bytes, chars, etc. in the Buffer are the same.
  - equals only compares part of Buffer, not every element in it is compared. Actually, it only compares the remaining elements in the Buffer.
- compareTo() method
  - The compareTo() method compares the remaining elements (byte, char, etc.) of the two Buffers and considers one Buffer to be "less than" the other if the following conditions are met:
    - The first unequal element is smaller than the corresponding element in the other Buffer.
    - All elements are equal, but the first buffer is exhausted before the other (the first buffer has fewer elements than the other).

Scatter/Gather

1. Java NIO began to support scatter/gather, scatter/gather is used to describe the operation of reading from or writing to Channel.
2. Scattering (scatter) reading from Channel refers to writing the read data into multiple buffers during a read operation. Therefore, the Channel "scatters" the data read from the Channel into multiple Buffers.
3. Gathering into a Channel means writing data from multiple buffers to the same Channel during a write operation. Therefore, the Channel “gathers” the data in multiple Buffers and sends them to the Channel.
4.scatter/gather is often used in situations where the transmitted data needs to be processed separately, such as transmitting a message consisting of a message header and a message body, you may scatter the message body and message header into different buffers, so that you can Convenient handling of message headers and message bodies.
5.Scattering Reads
- Scattering Reads means that data is read from one channel to multiple buffers.

ByteBuffer header = ByteBuffer.allocate(128);
ByteBuffer body   = ByteBuffer.allocate(1024);

ByteBuffer[] bufferArray = { header, body };

channel.read(bufferArray);

Note that the buffer is first inserted into the array, and then the array is used as the input parameter to channel.read(). The read() method writes the data read from the channel to the buffer in the order of the buffer in the array. When a buffer is full, the channel writes to another buffer.
Scattering Reads must fill the current buffer before moving the next buffer, which also means that it is not suitable for dynamic messages (Translator's Note: The message size is not fixed). In other words, if there is a message header and message body, the message header must be filled (eg 128byte) for Scattering Reads to work properly.
6.Gathering Writes
- Gathering Writes means that data is written from multiple buffers to the same channel.

ByteBuffer header = ByteBuffer.allocate(128);
ByteBuffer body   = ByteBuffer.allocate(1024);

//write data into buffers

ByteBuffer[] bufferArray = { header, body };

channel.write(bufferArray);

The buffers array is the input parameter of the write() method. The write() method will write data to the channel according to the order of the buffers in the array. Note that only the data between the position and the limit will be written. Therefore, if a buffer has a capacity of 128 bytes, but only contains 58 bytes of data, then the 58 bytes of data will be written to the channel. Therefore, in contrast to Scattering Reads, Gathering Writes can handle dynamic messages better.

data transfer between channels

1. In Java NIO, if one of the two channels is a FileChannel, you can directly transfer data from one channel to another channel.
2.transferFrom()
- The transferFrom() method of FileChannel can transfer data from the source channel to the FileChannel

RandomAccessFile fromFile = new RandomAccessFile("fromFile.txt", "rw");
FileChannel      fromChannel = fromFile.getChannel();

RandomAccessFile toFile = new RandomAccessFile("toFile.txt", "rw");
FileChannel      toChannel = toFile.getChannel();

long position = 0;
long count = fromChannel.size();

toChannel.transferFrom(position, count, fromChannel);

The input parameter position indicates that data is written to the target file from position, and count indicates the maximum number of bytes transferred. If the remaining space on the source channel is less than count bytes, the number of bytes transferred is less than the requested number of bytes.
Also note that in the implementation of SoketChannel, SocketChannel will only transmit data that is ready at the moment (maybe less than count bytes). Therefore, the SocketChannel may not transfer all the requested data (count bytes) into the FileChannel.
3.transferTo()
- The transferTo() method transfers data from FileChannel to other channels.

RandomAccessFile fromFile = new RandomAccessFile("fromFile.txt", "rw");
FileChannel      fromChannel = fromFile.getChannel();

RandomAccessFile toFile = new RandomAccessFile("toFile.txt", "rw");
FileChannel      toChannel = toFile.getChannel();

long position = 0;
long count = fromChannel.size();

fromChannel.transferTo(position, count, toChannel);

Selector

Selector is a component in Java NIO that can detect one or more NIO channels and know whether the channel is ready for events such as read and write. In this way, a single thread can manage multiple channels and thus multiple network connections.
The following is an example diagram of a single thread using one Selector to process 3 channels:
- 1. Why use Selector?
  - The benefit of using only a single thread to handle multiple Channels is that fewer threads are needed to handle the channels. In fact, all channels can be handled by only one thread. For the operating system, context switching between threads is expensive, and each thread occupies some resources of the system (such as memory). Therefore, the fewer threads used, the better.
- 2. Creation of Selector
  - Create a Selector by calling the Selector.open() method, as follows:
    - Selector selector = Selector.open();
- 3. Register the channel with the Selector
  - In order to use Channel and Selector together, the channel must be registered with the selector. It is implemented by the SelectableChannel.register() method, as follows:

channel.configureBlocking(false);
SelectionKey key = channel.register(selector,
    Selectionkey.OP_READ);

When used with Selector, Channel must be in non-blocking mode. This means that FileChannel cannot be used with Selector because FileChannel cannot be switched to non-blocking mode. And socket channels are fine.
Note the second parameter of the register() method. This is an "interest collection", meaning what events are of interest when listening to the Channel through the Selector. Four different types of events can be listened for:
- Connect
- Accept
- Read
- Write
Channel triggering an event means that the event is ready. Therefore, a channel successfully connecting to another server is called "connection ready". A server socket channel that is ready to receive incoming connections is called "ready to receive". A channel with data to read can be said to be "read-ready". A channel waiting to write data can be said to be "write ready".
These four events are represented by the four constants of SelectionKey:
- SelectionKey.OP_CONNECT
- SelectionKey.OP_ACCEPT
- SelectionKey.OP_READ
- SelectionKey.OP_WRITE
- If you are interested in more than one kind of event, you can use the "bitwise OR" operator to concatenate the constants, like this:
  - int interestSet = SelectionKey.OP_READ | SelectionKey.OP_WRITE;
4.SelectionKey
- interest collection
  - The interest set is the set of events of interest that you have selected. The interest collection can be read and written via the SelectionKey, like this:

int interestSet = selectionKey.interestOps();

boolean isInterestedInAccept  = (interestSet & SelectionKey.OP_ACCEPT) == SelectionKey.OP_ACCEPT；
boolean isInterestedInConnect = interestSet & SelectionKey.OP_CONNECT;
boolean isInterestedInRead    = interestSet & SelectionKey.OP_READ;
boolean isInterestedInWrite   = interestSet & SelectionKey.OP_WRITE;
//用“位与”操作interest 集合和给定的SelectionKey常量，可以确定某个确定的事件是否在interest 集合中。

ready collection
- The ready collection is the collection of operations for which the channel is ready. After a selection, you will first access the ready set. Selection will be explained in the next subsection. The ready collection can be accessed like this:int readySet = selectionKey.readyOps();
You can detect what events or actions are ready in the channel in the same way as you detect the interest collection. However, the following four methods are also available, all of which return a boolean type:

selectionKey.isAcceptable();
selectionKey.isConnectable();
selectionKey.isReadable();
selectionKey.isWritable();

Channel + Selector

Channel  channel  = selectionKey.channel();
Selector selector = selectionKey.selector();

attached object
- An object or more information can be attached to the SelectionKey so that a given channel can be easily identified. For example, you can attach a Buffer used with a channel, or some object that contains aggregated data. The method of use is as follows:

selectionKey.attach(theObject);
Object attachedObj = selectionKey.attachment();

You can also attach objects when registering the Channel with the Selector using the register() method.

SelectionKey key = channel.register(selector, SelectionKey.OP_READ, theObject);

Select channel by Selector
- Once one or more channels are registered with the Selector, several overloaded select() methods can be called. These methods return those channels that are ready for the event you are interested in (such as connect, accept, read, or write). In other words, if you are interested in "read-ready" channels, the select() method will return those channels that are ready for read events.
- Here is the select() method:
  - int select()//Block until at least one channel is ready on the event you registered.
  - int select(long timeout)//Same as select(), except that it will block the longest timeout milliseconds (parameter)
  - int selectNow()// does not block and returns immediately no matter what channel is ready (this method performs a non-blocking selection operation. If no channel has become selectable since the previous selection operation, this method returns zero directly. ).
- The int value returned by the select() method indicates how many channels are ready. That is, how many channels have become ready since the last time the select() method was called. If the select() method is called, it returns 1 because one of the channels becomes ready, and if the select() method is called again, it will return 1 again if another channel is ready. If nothing was done on the first ready channel, there are now two ready channels, but between each select() method call, only one channel is ready.
selectedKeys()
- Once the select() method is called and the return value indicates that one or more channels are ready, the ready channels in the "selected key set" can then be accessed by calling the selector's selectedKeys() method. As follows:

Set selectedKeys = selector.selectedKeys();

When registering a Channel as a Selector, the Channel.register() method returns a SelectionKey object. This object represents the channel registered with this Selector. These objects can be accessed through the SelectKey's selectedKeySet() method.
- This selected set of keys can be traversed to access ready channels.

Set selectedKeys = selector.selectedKeys();
Iterator keyIterator = selectedKeys.iterator();
while(keyIterator.hasNext()) {
    SelectionKey key = keyIterator.next();
    if(key.isAcceptable()) {
        // a connection was accepted by a ServerSocketChannel.
    } else if (key.isConnectable()) {
        // a connection was established with a remote server.
    } else if (key.isReadable()) {
        // a channel is ready for reading
    } else if (key.isWritable()) {
        // a channel is ready for writing
    }
    keyIterator.remove();
}

This loop goes through each key in the selected key set and detects the ready event for the channel corresponding to each key.
Note the keyIterator.remove() call at the end of each iteration. The Selector does not remove SelectionKey instances from the selected key set by itself. You must remove yourself when you are done processing the channel. The next time the channel becomes ready, the Selector will put it in the selected key set again.
The channel returned by the SelectionKey.channel() method needs to be converted to the type you want to deal with, such as ServerSocketChannel or SocketChannel.
wakeUp()
- A thread is blocked after calling the select() method, and there is a way for it to return from the select() method even if no channel is ready. Just let other threads call the Selector.wakeup() method on the object on which the first thread called the select() method. Threads blocked on the select() method will return immediately.
- If another thread calls the wakeup() method, but no thread is currently blocked on the select() method, the next thread that calls the select() method will "wake up" immediately.
close()
- Calling its close() method after using the Selector will close the Selector and invalidate all SelectionKey instances registered on the Selector. The channel itself does not close.
complete example

Selector selector = Selector.open();
channel.configureBlocking(false);
SelectionKey key = channel.register(selector, SelectionKey.OP_READ);
while(true) {
  int readyChannels = selector.select();
  if(readyChannels == 0) continue;
  Set selectedKeys = selector.selectedKeys();
  Iterator keyIterator = selectedKeys.iterator();
  while(keyIterator.hasNext()) {
    SelectionKey key = keyIterator.next();
    if(key.isAcceptable()) {
        // a connection was accepted by a ServerSocketChannel.
    } else if (key.isConnectable()) {
        // a connection was established with a remote server.
    } else if (key.isReadable()) {
        // a channel is ready for reading
    } else if (key.isWritable()) {
        // a channel is ready for writing
    }
    keyIterator.remove();
  }
}

FileChannel

A FileChannel in Java NIO is a channel that connects to a file. Files can be read and written through the file channel.
FileChannel cannot be set to non-blocking mode, it always runs in blocking mode.
1. Open FileChannel
- Before a FileChannel can be used, it must be opened. However, we cannot open a FileChannel directly, we need to obtain a FileChannel instance by using an InputStream, OutputStream or RandomAccessFile.

RandomAccessFile aFile = new RandomAccessFile("data/nio-data.txt", "rw");
FileChannel inChannel = aFile.getChannel();

2. Read data from FileChannel
- Call one of several read() methods to read data from the FileChannel. Such as:

ByteBuffer buf = ByteBuffer.allocate(48);
int bytesRead = inChannel.read(buf);

First, allocate a Buffer. Data read from FileChannel will be read into Buffer.
Then, call the FileChannel.read() method. This method reads data from FileChannel into Buffer. The int value returned by the read() method indicates how many bytes were read into the Buffer. If it returns -1, it means the end of the file is reached.
3. Write data to FileChannel
- Use the FileChannel.write() method to write data to the FileChannel. The parameter of this method is a Buffer.

String newData = "New String to write to file..." + System.currentTimeMillis();

ByteBuffer buf = ByteBuffer.allocate(48);
buf.clear();
buf.put(newData.getBytes());

buf.flip();

while(buf.hasRemaining()) {
    channel.write(buf);
}

Note that FileChannel.write() is called in a while loop. Because there is no guarantee how many bytes the write() method can write to the FileChannel at a time, the write() method needs to be called repeatedly until there are no more bytes in the Buffer that have not been written to the channel.
4. Close FileChannel
- You must close the FileChannel when you are done using it.

channel.close();

5. The position method of FileChannel
- Sometimes it may be necessary to read/write data in a specific location of the FileChannel. The current position of the FileChannel can be obtained by calling the position() method.
- The current position of the FileChannel can also be set by calling the position(long pos) method.

long pos = channel.position();
channel.position(pos +123);

If you set the position after the end-of-file character and then attempt to read data from the file channel, the read method will return -1 - the end-of-file flag.
If you set the position after the end-of-file character, and then write data to the channel, the file will be stretched to the current position and the data will be written. This can lead to "file holes", gaps between data written in physical files on disk.
6. The size method of FileChannel
- The size() method of a FileChannel instance will return the size of the file associated with the instance.

long fileSize = channel.size();

7. The truncate method of FileChannel
- A file can be truncated using the FileChannel.truncate() method. When intercepting a file, the part after the specified length in the file will be deleted.

channel.truncate(1024);
//这个例子截取文件的前1024个字节。

8. The force method of FileChannel
- The FileChannel.force() method forces the data in the channel that has not been written to disk to be written to disk. For performance reasons, the operating system caches data in memory, so there is no guarantee that data written to FileChannel will be written to disk immediately. To ensure this, the force() method needs to be called.
- The force() method has a boolean parameter that indicates whether to write file metadata (permission information, etc.) to disk at the same time.

channel.force(true);

Reprinted from Concurrent Programming Network – ifeve.com Link to this article: Java NIO Series Tutorials (7) FileChannel

JAVA NIO study notes - 01

NIO Profile

Buffer

Scatter/Gather

data transfer between channels

Selector

FileChannel

Guess you like