Author: Vivo Internet Server Team - Wang Ke
This article introduces the design and implementation of ManagedSelector and ExecutionStrategy in Jetty, and reveals Jetty's thread optimization ideas by comparing with the native select call. Jetty has designed an adaptive thread execution strategy (EatWhatYouKill), which uses the same thread to detect I/O events and process I/O events as much as possible without thread starvation, making full use of CPU cache and reducing thread switching s expenses. This optimization idea has certain reference significance for performance optimization in scenarios with a large number of I/O operations.
1. What is Jetty
Jetty is a web container like Tomcat, and its overall architecture is designed as follows:
Jetty generally consists of a series of Connectors, a series of Handlers and a ThreadPool.
Connector is Jetty's connector component. Compared with Tomcat's connector, Jetty's connector has its own characteristics in design.
Jetty's Connector supports the NIO communication model. The protagonist in the NIO model is the Selector. Jetty encapsulates its own Selector: ManagedSelector based on the Java native Selector.
2. Selector interaction in Jetty
2.1 Traditional Selector implementation
The conventional NIO programming idea is to use different threads to process the detection of I/O events and the processing of requests.
The specific process is:
-
start a thread;
-
Continuously call the select method in an infinite loop to detect the I/O status of the Channel;
-
Once the I/O event arrives, wrap the I/O event and some data into a Runnable;
-
Put the Runnable in a new thread for processing.
There are two threads working in this process: one is the I/O event detection thread and the other is the I/O event processing thread.
These two threads are in the relationship of " producer " and " consumer ".
The benefits of this design:
The advantage of using different threads to process the two jobs is that they do not interfere with each other and block each other.
Drawbacks of this design:
When the Selector detects the read-ready event, the data has been copied to the cache in the kernel, and the data is also in the cache of the CPU.
At this time, when the application program reads the data, if another thread is used to read it, it is very likely that the reading thread uses another CPU core instead of the previous CPU core that detects that the data is ready.
In this way, the data in the CPU cache will not be used, and thread switching also requires overhead.
2.2 ManagedSelector implementation in Jetty
Jetty's Connector puts the production and consumption of I/O events into the same thread.
If the thread is not blocked during execution, the operating system will use the same CPU core to execute the two tasks, which can not only make full use of the CPU cache, but also reduce the overhead of thread context switching.
ManagedSelector is essentially a Selector responsible for the detection and distribution of I/O events.
For ease of use, Jetty has made some extensions on the basis of Java's native Selector, and its member variables are as follows:
public class ManagedSelector extends ContainerLifeCycle implements Dumpable
{
// 原子变量，表明当前的ManagedSelector是否已经启动
private final AtomicBoolean _started = new AtomicBoolean(false);
// 表明是否阻塞在select调用上
private boolean _selecting = false;
// 管理器的引用，SelectorManager管理若干ManagedSelector的生命周期
private final SelectorManager _selectorManager;
// ManagedSelector的id
private final int _id;
// 关键的执行策略，生产者和消费者是否在同一个线程处理由它决定
private final ExecutionStrategy _strategy;
// Java原生的Selector
private Selector _selector;
// "Selector更新任务"队列
private Deque<SelectorUpdate> _updates = new ArrayDeque<>();
private Deque<SelectorUpdate> _updateable = new ArrayDeque<>();
...
}2.2.1 SelectorUpdate interface
Why do we need a "Selector update task" queue?
For Selector users, our operations on Selector are nothing more than registering Channel to Selector or telling Selector what I/O events I am interested in.
These operations are actually updates to the Selector state, and Jetty abstracts these operations into the SelectorUpdate interface.
/**
* A selector update to be done when the selector has been woken.
*/
public interface SelectorUpdate
{
void update(Selector selector);
}This means that Selector in ManagedSelector cannot be operated directly, but a task class needs to be submitted to ManagedSelector.
This class needs to implement the update method of the SelectorUpdate interface, and define the operations to be done on the ManagedSelector in the update method.
For example, the Endpoint component in Connector is interested in read-ready events.
It submits an internal task class ManagedSelector.SelectorUpdate to ManagedSelector:
_selector.submit(_updateKeyAction);This _updateKeyAction is a SelectorUpdate instance, and its update method is implemented as follows:
private final ManagedSelector.SelectorUpdate _updateKeyAction = new ManagedSelector.SelectorUpdate()
{
@Override
public void update(Selector selector)
{
// 这里的updateKey其实就是调用了SelectionKey.interestOps(OP_READ);
updateKey();
}
};In the update method, the interestOps method of the SelectionKey class is called, and the parameter passed in is OP_READ, which means that I am interested in the read-ready event on this Channel.
2.2.2 Selectable interface
With the update method above, who will execute these updates? The answer is ManagedSelector itself.
It fetches these SelectorUpdate tasks one by one in an infinite loop.
When an I/O event arrives, ManagedSelector uses a task class interface (Selectable interface) to determine which function handles the event.
public interface Selectable
{
// 当某一个Channel的I/O事件就绪后，ManagedSelector会调用的回调函数
Runnable onSelected();
// 当所有事件处理完了之后ManagedSelector会调的回调函数
void updateKey();
}The onSelected() method of the Selectable interface returns a Runnable, which is the corresponding processing logic when the I/O event is ready.
When ManagedSelector detects that the I/O event on a Channel is ready, ManagedSelector calls the onSelected method of the class bound to the Channel to get a Runnable.
Then throw the Runnable to the thread pool for execution.
3. Jetty's thread optimization ideas
3.1 Implementation of ExecutionStrategy in Jetty
The usage interaction of ManagedSelector was introduced earlier:
-
How to register Channel and I/O events
-
What kind of processing class is provided to handle I/O events
So how ManagedSelector manages and maintains the set of channels registered by users in a unified way? The answer is the ExecutionStrategy interface.
This interface entrusts the production of specific tasks to the internal interface Producer, and implements specific execution logic in its own produce method.
The task of this Runnable can be executed by the current thread, or it can be executed in a new thread.
public interface ExecutionStrategy
{
// 只在HTTP2中用到的一个方法，暂时忽略
public void dispatch();
// 实现具体执行策略，任务生产出来后可能由当前线程执行，也可能由新线程来执行
public void produce();
// 任务的生产委托给Producer内部接口
public interface Producer
{
// 生产一个Runnable（任务）
Runnable produce();
}
}Implement the Produce interface to produce tasks. Once the tasks are produced, ExecutionStrategy will be responsible for executing the tasks.
private class SelectorProducer implements ExecutionStrategy.Producer
{
private Set<SelectionKey> _keys = Collections.emptySet();
private Iterator<SelectionKey> _cursor = Collections.emptyIterator();
@Override
public Runnable produce()
{
while (true)
{
// 如果Channel集合中有I/O事件就绪，调用前面提到的Selectable接口获取Runnable,直接返回给ExecutionStrategy去处理
Runnable task = processSelected();
if (task != null)
return task;
// 如果没有I/O事件就绪，就干点杂活，看看有没有客户提交了更新Selector的任务，就是上面提到的SelectorUpdate任务类。
processUpdates();
updateKeys();
// 继续执行select方法，侦测I/O就绪事件
if (!select())
return null;
}
}
}SelectorProducer is an inner class of ManagedSelector.
SelectorProducer implements the produce method in the Producer interface in ExecutionStrategy and needs to return a Runnable to ExecutionStrategy.
In the produce method, SelectorProducer mainly does three things:
-
If there is an I/O event ready in the Channel collection, call the aforementioned Selectable interface to get the Runnable, and return it directly to ExecutionStrategy for processing.
-
If there is no I/O event ready, do some chores to see if any customer has submitted a task to update the event registration on the Selector, which is the SelectorUpdate task class mentioned above.
-
After finishing the chores, continue to execute the select method to detect I/O ready events.
3.2 Jetty's thread execution strategy
3.2.1 ProduceConsume(PC) thread execution strategy
Task producers produce and execute tasks in turn, corresponding to the NIO communication model is to use a thread to detect and process all I/O events on a ManagedSelector.
The subsequent I/O events have to wait for the previous I/O events to be processed, which is obviously not efficient.
In the figure, green represents the production of a task, and blue represents the execution of this task, the same below.
3.2.2 ProduceExecuteConsume (PEC) thread execution strategy
The task producer starts a new thread to execute the task, which is a typical I/O event detection and processing with different threads.
The disadvantage is that the CPU cache cannot be utilized, and the cost of thread switching is high.
In the figure, brown represents thread switching, the same below.
3.2.3 ExecuteProduceConsume (EPC) thread execution strategy
The task producer runs the task by itself, which may create a new thread to continue producing and executing the task.
Its advantage is that it can utilize the CPU cache, but the potential problem is that if the execution time of the business code that processes I/O events is too long, it will cause a large number of threads to be blocked and thread starvation.
3.2.4 EatWhatYouKill (EWYK) improved thread execution strategy
This is Jetty's improvement of the ExecuteProduceConsume strategy, which is equivalent to ExecuteProduceConsume when the thread pool thread is sufficient;
When the system is busy and there are not enough threads, switch to the ProduceExecuteConsume strategy.
The reason for this is:
ExecuteProduceConsume executes the production and consumption of I/O events in the same thread. The threads it uses come from Jetty's global thread pool. These threads may be blocked by business code. If there are too many blocks, the threads in the global thread pool will naturally not be enough. In the worst case, there is no thread available for I/O event detection, which will cause the Connector to reject the browser request.
So Jetty made an optimization :
In the case of low threads, the ProduceExecuteConsume strategy is implemented, and I/O detection is processed by a dedicated thread, and the processing of I/O events is thrown to the thread pool for processing. In fact, it is placed in the thread pool queue for slow processing.
Four. Summary
This article introduces the design and implementation of ManagedSelector and ExecutionStrategy based on Jetty-9, and introduces the differences between the three thread execution strategies of PC, PEC, and EPC. It can be seen from the improvement of Jetty's thread execution strategy that Jetty's thread execution strategy will take priority Using EPC enables production and consumption tasks to run on the same thread, which can make full use of hot caches and avoid scheduling delays.
This also provides us with some ideas for performance optimization:
-
In the case of ensuring that thread starvation does not occur, try to use the same thread for production and consumption to make full use of the CPU cache and reduce the overhead of thread switching.
-
Choose the most suitable execution strategy according to the actual scenario. By combining multiple sub-strategies, you can also maximize strengths and avoid weaknesses to achieve the effect of 1+1>2.
Reference documents: