1 Introduction
In the traditional blocking IO ( BIO ) before JDK1.4, the server needs to create a separate thread for each client connection to serve it. Since JDK1.4, NIO non-blocking IO has appeared, and it only needs a single thread. Receive multiple client requests, and the details of actually processing each request can be completed efficiently using multi-threading. These processing threads are separated from the specific business logic to achieve IO multiplexing.
2. Source code analysis
First start with a typical NIO usage code:
Selector selector = Selector.open();
ServerSocketChannel ssc = ServerSocketChannel.open();
ssc.configureBlocking(false);
ssc.socket().bind(new InetSocketAddress(9527));
ssc.register(selector, SelectionKey.OP_ACCEPT);
while(true){
int n = selector.select();
if (n <= 0) continue;
Iterator it = selector.selectedKeys().iterator();
while(it.hasNext()){
SelectionKey key = (SelectionKey)it.next();
if (key.isAcceptable()){
SocketChannel sc= ((ServerSocketChannel) key.channel()).accept();
sc.configureBlocking(false);
sc.register(key.selector(), SelectionKey.OP_READ|SelectionKey.OP_WRITE);
}
if (key.isReadable()){
SocketChannel channel = ((SocketChannel) key.channel());
ByteBuffer bf = ByteBuffer.allocate(10);
int read = channel.read(bf);
System.out.println("read "+read+" : "+new String(bf.array()).trim());
}
if (key.isWritable()){
SocketChannel channel = ((SocketChannel) key.channel());
channel.write(ByteBuffer.wrap(new String("hello client").getBytes()));
}
it.remove();
}
}
2.1 Selector.open() Get the selector.
public static Selector open() throws IOException {
return SelectorProvider.provider().openSelector();
}
public static SelectorProvider provider() {
synchronized (lock) {
if (provider != null)
return provider;
return AccessController.doPrivileged(
new PrivilegedAction<SelectorProvider>() {
public SelectorProvider run() {
if (loadProviderFromProperty())
return provider;
if (loadProviderAsService())
return provider;
provider = sun.nio.ch.DefaultSelectorProvider.create();
return provider;
}
});
}
}
As you can see from the Selector source code, the open method is handed over to the selectorProvider for processing. Among them, provider = sun.nio.ch.DefaultSelectorProvider.create(); will return different implementation classes according to the operating system, and the windows platform will return WindowsSelectorProvider; the Linux platform will choose whether to use select/poll mode or epoll mode according to different kernel versions .
public static SelectorProvider create() {
PrivilegedAction pa = new GetPropertyAction("os.name");
String osname = (String) AccessController.doPrivileged(pa);
if ("SunOS".equals(osname)) {
return new sun.nio.ch.DevPollSelectorProvider();
}
// use EPollSelectorProvider for Linux kernels >= 2.6
if ("Linux".equals(osname)) {
pa = new GetPropertyAction("os.version");
String osversion = (String) AccessController.doPrivileged(pa);
String[] vers = osversion.split("\\.", 0);
if (vers.length >= 2) {
try {
int major = Integer.parseInt(vers[0]);
int minor = Integer.parseInt(vers[1]);
if (major > 2 || (major == 2 && minor >= 6)) {
return new sun.nio.ch.EPollSelectorProvider();
}
} catch (NumberFormatException x) {
// format not recognized
}
}
}
return new sun.nio.ch.PollSelectorProvider();
}
sun.nio.ch.EPollSelectorProvider
public AbstractSelector openSelector() throws IOException {
return new EPollSelectorImpl(this);
}
sun.nio.ch.PollSelectorProvider
public AbstractSelector openSelector() throws IOException {
return new PollSelectorImpl(this);
}
It can be seen that if the Linux kernel version is >= 2.6, the specific SelectorProvider is EPollSelectorProvider, otherwise it is the default PollSelectorProvider. In fact, this update is only after JDK5U9.
public static SelectorProvider create() {
return new sun.nio.ch.WindowsSelectorProvider();
}
sun.nio.ch.WindowsSelectorProvider
public AbstractSelector openSelector() throws IOException {
return new WindowsSelectorImpl(this);
}
WindowsSelectorImpl(SelectorProvider sp) throws IOException {
super(sp);
pollWrapper = new PollArrayWrapper(INIT_CAP);
wakeupPipe = Pipe.open();
wakeupSourceFd = ((SelChImpl)wakeupPipe.source()).getFDVal();
// Disable the Nagle algorithm so that the wakeup is more immediate
SinkChannelImpl sink = (SinkChannelImpl)wakeupPipe.sink();
(sink.sc).socket().setTcpNoDelay(true);
wakeupSinkFd = ((SelChImpl)sink).getFDVal();
pollWrapper.addWakeupSocket(wakeupSourceFd, 0);
}
void addWakeupSocket(int fdVal, int index) {
putDescriptor(index, fdVal);
putEventOps(index, POLLIN);
}
Next, the analysis is performed based on the implementation of Windows. In the process of instantiating WindowsSelectorImpl in the openSelector method,
1). The PollWrapper is instantiated, and pollWrapper uses the Unsafe class to apply for a piece of physical memory, which is used to store the socket handle fdVal during registration and the event data structure pollfd.
2) Pipe.open() opens a pipe (the implementation of opening the pipe will be seen later); get two file descriptors wakeupSourceFd and wakeupSinkFd; put the file descriptor (wakeupSourceFd) of the wakeup end into pollWrapper. The addWakeupSocket method adds the source The POLLIN event ( with data to read ) is marked as interesting. When there is data written on the sink side, the file description corresponding to the source will be in the ready state.
public static Pipe open() throws IOException {
return SelectorProvider.provider().openPipe();
}
public Pipe openPipe() throws IOException {
return new PipeImpl(this);
}
PipeImpl(final SelectorProvider sp) throws IOException {
try {
AccessController.doPrivileged(new Initializer(sp));
} catch (PrivilegedActionException x) {
throw (IOException)x.getCause();
}
}
private Initializer(SelectorProvider sp) {
this.sp = sp;
}
public Void run() throws IOException {
LoopbackConnector connector = new LoopbackConnector();
connector.run();
....//omit
}
private class LoopbackConnector implements Runnable {
@Override
public void run() {
ServerSocketChannel ssc = null;
SocketChannel sc1 = null;
SocketChannel sc2 = null;
try {
// Loopback address
InetAddress lb = InetAddress.getByName("127.0.0.1");
assert(lb.isLoopbackAddress());
InetSocketAddress at = null;
for(;;) {
// Bind ServerSocketChannel to a port on the loopback
// address
if (ssc == null || !ssc.isOpen()) {
ssc = ServerSocketChannel.open ();
ssc.socket().bind(new InetSocketAddress(lb, 0));
sa = new InetSocketAddress(lb, ssc.socket().getLocalPort());
}
// Establish connection (assume connections are eagerly
// accepted)
sc1 = SocketChannel.open(sa);
ByteBuffer bb = ByteBuffer.allocate(8);
long secret = rnd.nextLong();
bb.putLong(secret).flip();
sc1.write(bb);
// Get a connection and verify it is legitimate
sc2 = ssc.accept();
bb.clear();
sc2.read(bb);
bb.rewind();
if (bb.getLong() == secret)
break;
sc2.close();
sc1.close();
}
// Create source and sink channels
source = new SourceChannelImpl(sp, sc1);
sink = new SinkChannelImpl(sp, sc2);
} catch (IOException e) {
try {
if (sc1 != null)
sc1.close();
if (sc2 != null)
sc2.close();
} catch (IOException e2) {}
yes = e;
} finally {
try {
if (ssc != null)
ssc.close();
} catch (IOException e2) {}
}
}
}
}
Through the code analysis of creating a pipeline: the specific implementation of creating a pipeline is also closely related to the specific operating system. Here, taking Windows as an example, a PipeImpl object is created. After AccessController.doPrivileged is called, the run method of the initializer will be executed immediately. In the run method, the implementation under Windows is to create two local socketChannels, and then connect them (the linking process is performed by writing a random long to check the connection of the two sockets). The two socketChannels implement the source and sink ends of the pipeline respectively. By consulting the information, under Linux, it is directly using the pipes provided by the operating system.
At this point, Selector.open() is completed. To sum up, the following things are mainly completed:
1. Instantiate the pollWrapper object, which is used to store the socket handle fdVal during registration and the event data structure pollfd in the future.
2. According to different operating systems, the pipes for self-awakening are implemented. Windows creates a pair of socket channels connected to itself, and Linux directly uses the pipes provided by the system. At the same time, according to different kernel versions of linux, different mechanisms for event notification at the bottom layer, select/poll or epoll, will also be selected.
2.2 serverSocketChannel.register(selector, SelectionKey.OP_ACCEPT); channel registration
public final SelectionKey register(Selector sel, int ops, Object att)
throws ClosedChannelException{
synchronized (regLock) {
SelectionKey k = findKey(sel);
if (k != null) {
k.interestOps(ops);
k.attach (att);
}
if (k == null) {
// New registration
synchronized (keyLock) {
if (!isOpen())
throw new ClosedChannelException();
k = ((AbstractSelector)sel).register(this, ops, att);
addKey(k);
}
}
return k;
}
}
If the channel and selector are already registered, add events and attachments directly. Otherwise, the registration process is implemented through the selector.
protected final SelectionKey register(AbstractSelectableChannel ch,
int ops, Object attachment) {
if (!(ch instanceof SelChImpl))
throw new IllegalSelectorException();
SelectionKeyImpl k = new SelectionKeyImpl((SelChImpl)ch, this);
k.attach(attachment);
synchronized (publicKeys) {
implRegister(k);
}
k.interestOps(ops);
return k;
}
protected void implRegister(SelectionKeyImpl ski) {
synchronized (closeLock) {
if (pollWrapper == null)
throw new ClosedSelectorException();
growIfNeeded();
channelArray[totalChannels] = ski;
ski.setIndex(totalChannels);
fdMap.put(ski);
keys.add(ski);
pollWrapper.addEntry(totalChannels, ski);
totalChannels++;
}
}
private void growIfNeeded() {
if (channelArray.length == totalChannels) {
int newSize = totalChannels * 2; // Make a larger array
SelectionKeyImpl temp[] = new SelectionKeyImpl[newSize];
System.arraycopy(channelArray, 1, temp, 1, totalChannels - 1);
channelArray = temp;
pollWrapper.grow(newSize);
}
if (totalChannels % MAX_SELECTABLE_FDS == 0) { // more threads needed
pollWrapper.addWakeupSocket(wakeupSourceFd, totalChannels);
totalChannels ++;
threadsCount++;
}
}
void addEntry(int index, SelectionKeyImpl ski) {
putDescriptor(index, ski.channel.getFDVal());
}
The process of registering through the selector mainly completes the following things:
- Initialize the SelectionKeyImpl object with the current channel and selector as parameters, and add the attachment attachment.
- If the current number of channels totalChannels is equal to the size of the SelectionKeyImpl array, expand the SelectionKeyImpl array and pollWrapper.
- If totalChannels % MAX_SELECTABLE_FDS == 0, open one more thread to process the selector. The select system call on windows has a maximum file descriptor limit, and only 1024 file descriptors can be polled at a time. If there are more than 1024, multi-threaded polling is required.
- ski.setIndex(totalChannels) selects the key to record the index position in the array.
- keys.add(ski); Add the selection key to the set of registered keys.
- fdMap.put(ski); Save the mapping relationship between the file descriptor corresponding to the selection key and the selection key.
- pollWrapper.addEntry will add the socket handle in selectionKeyImpl to the corresponding pollfd.
- The k.interestOps(ops) method will also eventually add the event to the corresponding pollfd.
2.3 selector.select();
public int select() throws IOException {
return select(0);
}
public int select(long timeout) throws IOException
{
if (timeout < 0)
throw new IllegalArgumentException("Negative timeout");
return lockAndDoSelect((timeout == 0) ? -1 : timeout);
}
private int lockAndDoSelect(long timeout) throws IOException {
synchronized (this) {
if (!isOpen())
throw new ClosedSelectorException();
synchronized (publicKeys) {
synchronized (publicSelectedKeys) {
return doSelect(timeout);
}
}
}
}
当调用selector.select()以及select(0)时，JDK对参数进行修正，其实传给
doSelect
的timeout为-1。当调用的是selectNow()的时候，timeout则为0，直接以负数作为参数则会抛出异常，
其中的doSelector又回到我们的Windows实现：
protected int doSelect(long timeout) throws IOException {
if (channelArray == null)
throw new ClosedSelectorException();
this.timeout = timeout; // set selector timeout
processDeregisterQueue();
if (interruptTriggered) {
resetWakeupSocket();
return 0;
}
// Calculate number of helper threads needed for poll. If necessary
// threads are created here and start waiting on startLock
adjustThreadsCount();
finishLock.reset(); // reset finishLock
// Wakeup helper threads, waiting on startLock, so they start polling.
// Redundant threads will exit here after wakeup.
startLock.startThreads();
// do polling in the main thread. Main thread is responsible for
// first MAX_SELECTABLE_FDS entries in pollArray.
try {
begin();
try {
subSelector.poll();
} catch (IOException e) {
finishLock.setException(e); // Save this exception
}
// Main thread is out of poll(). Wakeup others and wait for them
if (threads.size() > 0)
finishLock.waitForHelperThreads();
} finally {
end();
}
// Done with poll(). Set wakeupSocket to nonsignaled for the next run.
finishLock.checkForException();
processDeregisterQueue();
int updated = updateSelectedKeys();
// Done with poll(). Set wakeupSocket to nonsignaled for the next run.
resetWakeupSocket();
return updated;
}
private int poll() throws IOException{ // poll for the main thread
return poll0(pollWrapper.pollArrayAddress,
Math.min(totalChannels, MAX_SELECTABLE_FDS),
readFds, writeFds, exceptFds, timeout);
}
The processDeregisterQueue
method is mainly to process the canceled key set. By calling the cancel() method, the selection key is added to the canceled key set. This method will remove the corresponding channel from the channelArray and adjust the number of channels and threads. Remove selection keys from map and keys, remove selection keys on channel and close channel. At the same time, it is also found that this method is called before and after the poll method is called, which is to ensure that the keys cancelled during the period of time when the poll method is blocked can be properly handled and cleaned up in time.
The adjustThreadsCount method is similar to the previous thread number adjustment, and adjusts the number of threads according to the file descriptor limit of the maximum select operation of the operating system.
subSelector.poll() is the core of select, which is implemented by the native function poll0, and passes pollWrapper.pollArrayAddress as a parameter to poll0. The readFds, writeFds and exceptFds arrays are used to store the results of the underlying select. The first position of the array is to store The total number of sockets where the event occurred, and the rest of the locations store the socket handle fd where the event occurred.
WindowsSelectorImpl.c
----
Java_sun_nio_ch_WindowsSelectorImpl_00024SubSelector_poll0(JNIEnv *env, jobject this,
jlong pollAddress, jint numfds,
jintArray returnReadFds, jintArray returnWriteFds,
jintArray returnExceptFds, jlong timeout)
{
static struct timeval zerotime = {0, 0};
if (timeout == 0) {
tv = &zerotime;
} else if (timeout < 0) {
tv = NULL;
} else {
tv = &timevalue;
tv->tv_sec = (long)(timeout / 1000);
tv->tv_usec = (long)((timeout % 1000) * 1000);
}
// 代码.... 此处省略
/* Call select */
if ((result = select(0 , &readfds, &writefds, &exceptfds, tv))
== SOCKET_ERROR) {
/* Bad error - this should not happen frequently */
/* Iterate over sockets and call select() on each separately */
// 代码.... 此处省略
for (i = 0; i < numfds; i++) {
/* prepare select structures for the i-th socket */
// 代码.... 此处省略
/* call select on the i-th socket */
if (select(0, &errreadfds, &errwritefds, &errexceptfds, &zerotime)
== SOCKET_ERROR) {
//代码....此处省略
}
}
}
}
通过这一段调用C语言的poll0实现(这段代码主要意义在于调用了select函数，其他逻辑只是针对发生SOCKET_ERROR错误的时候，对每一个socket进行了单独的select调用)，我们可以看到，Windows调用了底层的select函数，这里的select就是轮询pollArray中的FD，看有没有事件发生，如果有事件发生收集所有发生事件的FD，退出阻塞。当调用selector.select()以及select(0)时，JDK对参数进行修正，其实传给底层poll0的timeout为-1。当调用的是selectNow()的时候，timeout则为0，直接以负数作为参数则会抛出异常，当传给底层select的参数tv为0时立即返回，为NULL时将会无限期阻塞直到事件发生。
最后一步调用updateSelectedKeys。这个方法完成了选择键的更新，具体实现：
private int updateSelectedKeys() {
updateCount++;
int numKeysUpdated = 0;
numKeysUpdated += subSelector.processSelectedKeys(updateCount);
for (SelectThread t: threads) {
numKeysUpdated += t.subSelector.processSelectedKeys(updateCount);
}
return numKeysUpdated;
}
//以上对主线程和各个helper线程(因为最大文件句柄数限制作出线程调整创建的线程)都调用了
processSelectedKeys方法。
private int processSelectedKeys(long updateCount) {
int numKeysUpdated = 0;
numKeysUpdated += processFDSet(updateCount, readFds,
PollArrayWrapper.POLLIN,
false);
numKeysUpdated += processFDSet(updateCount, writeFds,
PollArrayWrapper.POLLCONN |
PollArrayWrapper.POLLOUT,
false);
numKeysUpdated += processFDSet(updateCount, exceptFds,
PollArrayWrapper.POLLIN |
PollArrayWrapper.POLLCONN |
PollArrayWrapper.POLLOUT,
true);
return numKeysUpdated;
}
//processSelectedKeys方法分别对读选择键集、写选择键集，异常选择键集调用了processFDSet方法
private int processFDSet(long updateCount, int[] fds, int rOps,
boolean isExceptFds)
{
int numKeysUpdated = 0;
for (int i = 1; i <= fds[0]; i++) {
int desc = fds[i];
if (desc == wakeupSourceFd) {
synchronized (interruptLock) {
interruptTriggered = true;
}
continue;
}
MapEntry me = fdMap.get(desc);
// If me is null, the key was deregistered in the previous
// processDeregisterQueue.
if (me == null)
continue;
SelectionKeyImpl sk = me.ski;
// The descriptor may be in the exceptfds set because there is
// OOB data queued to the socket. If there is OOB data then it
// is discarded and the key is not added to the selected set.
if (isExceptFds &&
(sk.channel() instanceof SocketChannelImpl) &&
discardUrgentData(desc))
{
continue;
}
if (selectedKeys.contains(sk)) { // Key in selected set
if (me.clearedCount != updateCount) {
if (sk.channel.translateAndSetReadyOps(rOps, sk) &&
(me.updateCount != updateCount)) {
me.updateCount = updateCount;
numKeysUpdated++;
}
} else { // The readyOps have been set; now add
if (sk.channel.translateAndUpdateReadyOps(rOps, sk) &&
(me.updateCount != updateCount)) {
me.updateCount = updateCount;
numKeysUpdated++;
}
}
me.clearedCount = updateCount;
} else { // Key is not in selected set yet
if (me.clearedCount != updateCount) {
sk.channel.translateAndSetReadyOps(rOps, sk);
if ((sk.nioReadyOps() & sk.nioInterestOps()) != 0) {
selectedKeys.add(sk);
me.updateCount = updateCount;
numKeysUpdated++;
}
} else { // The readyOps have been set; now add
sk.channel.translateAndUpdateReadyOps(rOps, sk);
if ((sk.nioReadyOps() & sk.nioInterestOps()) != 0) {
selectedKeys.add(sk);
me.updateCount = updateCount;
numKeysUpdated++;
}
}
me.clearedCount = updateCount;
}
}
return numKeysUpdated;
}
通过以上代码分析：
1、忽略wakeupSourceFd，这个文件描述符用于唤醒用的，与用户具体操作无关，所以忽略；
2、过滤fdMap中不存在的文件描述符，因为已被注销；
3、忽略oob data(搜了一下：out of band data指带外数据，有时也称为加速数据， 是指连接双方中的一方发生重要事情，想要迅速地通知对方 )，这也不是用户关心的；
4、如果通道的键还没有处于已选择的键的集合中，那么键的ready集合将被清空，然后表示操作系统发现的当前通道已经准备好的操作的比特掩码将被设置；
5、如果键在已选择的键的集合中。操作系统发现的当前已经准备好的操作的比特掩码将会被更新进ready集合，而对已经存在的任何结果集不做清除处理。
2.4 wakeup
public Selector wakeup() {
synchronized (interruptLock) {
if (!interruptTriggered) {
setWakeupSocket();
interruptTriggered = true;
}
}
return this;
}
// Sets Windows wakeup socket to a signaled state.
private void setWakeupSocket() {
setWakeupSocket0(wakeupSinkFd);
}
private native void setWakeupSocket0(int wakeupSinkFd);
//WindowsSelectorImpl.c
JNIEXPORT void JNICALL
Java_sun_nio_ch_WindowsSelectorImpl_setWakeupSocket0(JNIEnv *env, jclass this,
jint scoutFd)
{
/* Write one byte into the pipe */
send(scoutFd, (char*)&POLLIN, 1, 0);
}
如果线程正阻塞在select方法上，调用wakeup方法会使阻塞的选择操作立即返回，通过以上Windows的实现其实是向pipe的sink端写入了一个字节，source文件描述符就会处于就绪状态，poll方法会返回，从而导致select方法返回。而在其他solaris或者linux系统上其实采用系统调用pipe来完成管道的创建，相当于直接用了系统的管道。通过以上代码还可以看出，调用wakeup设置了interruptTriggered的标志位，所以连续多次调用wakeup的效果等同于一次调用。