Eureka series (06) message broadcasting (under): TaskDispacher of Acceptor - Worker mode

Eureka series (06) message broadcasting (under): TaskDispacher of Acceptor - Worker mode

[TOC]

0. the Spring Cloud Series catalog - Eureka articles

Eureka message broadcast to explain the main three parts:

1. Server List Management: PeerEurekaNodes manage all PeerEurekaNode nodes.
2. News broadcast mechanism Analysis: After PeerAwareInstanceRegistryImpl receive messages to clients, the first step: to update the local registration information; the second step: through all PeerEurekaNode, forwarded to other nodes.
3. TaskDispacher message processing: Acceptor - Worker Mode.

First, review the process messages broadcast on a Eureka series (05) news broadcast (on): message broadcast principle analysis of the source code of Eureka news broadcasts were analyzed, PeerAwareInstanceRegistryImpl message broadcasting task entrusted to PeerEurekaNode. Internal PeerEurekaNode using TaskDispacher of Acceptor - Worker mode asynchronous processing. This paper focuses on the asynchronous processing mechanism.

1. Acceptor - Worker mode principle

Figure 1: Acceptor - Worker mode principle

sequenceDiagram participant TaskDispatcher participant AcceptorExecutor participant WorkerRunnable note left of TaskDispatcher : 接收消息广播任务<br/>process TaskDispatcher ->> AcceptorExecutor : process WorkerRunnable ->> AcceptorExecutor : requestWorkItem WorkerRunnable ->> AcceptorExecutor : requestWorkItems note right of WorkerRunnable : run opt error WorkerRunnable -->> AcceptorExecutor : reprocess end

Summary: TaskDispatcher receive messages broadcast mission, the actual AcceptorExecutor thread, after the execution by the WorkerRunnable thread. WorkerRunnable thread execution logic has been analyzed, the following look at AcceptorExecutor source.

AcceptorExecutor main methods:

• process/reprocessBroadcasting task receives the message, stored in acceptorQueue/reprocessQueuethe queue.
• requestWorkItem/requestWorkItemsGets the message broadcast tasks to be performed singleItemWorkQueue/batchWorkQueue.

2. AcceptorExecutor

Figure 2: AcceptorRunner tasking

graph LR A(Client) B(reprocessQueue) C(acceptorQueue) D(pendingTasks<br/>processingOrder) E(batchWorkQueue) F(singleItemWorkQueue) A -- reprocess --> B A -- process --> C B -- drainReprocessQueue --> D C -- drainAcceptorQueue --> D D -- assignBatchWork --> E D -- assignSingleItemWork --> F

Summary: AcceptorRunner threads per 10s polls, message broadcast from the task acceptorQueue -> pendingTasks -> batchWorkQueueexecution, WorkerRunner direct access batchWorkQueue task execution thread.

If pendingTasks task overload (default 10000) dropped principle is: First, discard the oldest tasks and retry tasks, task execution date. The second is to perform the same task taskId only the latest task .

• After pendingTasks queue is full, reprocessQueue task will be fully discarded, acceptorQueue discard the oldest task execution date of the task.
• AcceptorExecutor in PeerEurekaNode initialization process. PendingTasks default maximum number of tasks maxBufferSize = 10000 Ge, a batch of maximum number of maxBatchingSize = 200 Ge, a batch of the maximum delay time is maxBatchingDelay = 500ms.

2.1 Properties

Internal AcceptorExecutor have multiple queues, maintenance tasks, functions queues as follows:

private final int maxBufferSize;		// pendingTasks队列最大值，默认值 10000
private final int maxBatchingSize;		// 一次批处理的最大任务数，默认值 250
private final long maxBatchingDelay;	// 任务的最大延迟时间，默认值 500ms

// 1. 接收消息广播的任务
private final BlockingQueue<TaskHolder<ID, T>> acceptorQueue;
private final BlockingDeque<TaskHolder<ID, T>> reprocessQueue;

// 2. 默认每 10s 轮询一次，将接收的消息处理一次
//    AcceptorRunner 单线程处理，所以是普通队列
private final Map<ID, TaskHolder<ID, T>> pendingTasks;
private final Deque<ID> processingOrder;

// 3. 即将要处理的消息广播任务
private final Semaphore singleItemWorkRequests;
private final BlockingQueue<TaskHolder<ID, T>> singleItemWorkQueue;

private final Semaphore batchWorkRequests;
private final BlockingQueue<List<TaskHolder<ID, T>>> batchWorkQueue;

2.2 source code analysis

2.2.1 AcceptorRunner the overall process

class AcceptorRunner implements Runnable {
    @Override
    public void run() {
        long scheduleTime = 0;
        while (!isShutdown.get()) {
            try {
                // 1. 将任务从 reprocessQueue/acceptorQueue -> pendingTasks
                drainInputQueues();
                int totalItems = processingOrder.size();
                long now = System.currentTimeMillis();
                // 2. trafficShaper 流量整行，执行失败后的延迟时间
                //    congestionRetryDelayMs 100ms 执行一次
                //    networkFailureRetryMs 1000ms 执行一次
                if (scheduleTime < now) {
                    scheduleTime = now + trafficShaper.transmissionDelay();
                }
                // 3. pendingTasks -> batchWorkQueue
                if (scheduleTime <= now) {
                    assignBatchWork();
                    assignSingleItemWork();
                }
                // 4. 没有可执行的任务了，等待 10s
                if (totalItems == processingOrder.size()) {
                    Thread.sleep(10);
                }
            } catch (InterruptedException ex) {
            } catch (Throwable e) {
            }
        }
    }
}

Summary: AcceptorRunner threads per 10s polls, message broadcast from the task acceptorQueue -> pendingTasks -> batchWorkQueueexecution, WorkerRunner direct access batchWorkQueue task execution thread.

• drainInputQueues Tasks from acceptorQueue -> pendingTasks
• assignBatchWork/assignSingleItemWork Tasks from pendingTasks -> batchWorkQueue

2.2.2 drainInputQueues

drainInputQueues method of processing tasks reprocessQueue / acceptorQueue queue to pendingTasks, only to pendingTasks the task will be processed, otherwise discarded.

// 先处理 reprocessQueue，再处理 acceptorQueue。
// 默认 acceptorQueue 会覆盖 reprocessQueue 中的任务，也就是最新的任务会覆盖重试的任务
private void drainInputQueues() throws InterruptedException {
    do {
        drainReprocessQueue();	// reprocessQueue
        drainAcceptorQueue();	// acceptorQueue

        if (!isShutdown.get()) {
            if (reprocessQueue.isEmpty() && acceptorQueue.isEmpty() && pendingTasks.isEmpty()) {
                TaskHolder<ID, T> taskHolder = acceptorQueue.poll(10, TimeUnit.MILLISECONDS);
                if (taskHolder != null) {
                    appendTaskHolder(taskHolder);
                }
            }
        }
    } while (!reprocessQueue.isEmpty() || !acceptorQueue.isEmpty() || pendingTasks.isEmpty());
}

Summary: drainInputQueues method message received from the broadcast tasks reprocessQueue / acceptorQueue -> pendingTasks. If the principle task execution is mostly pendingTasks: discards the oldest of tasks and retry tasks, task execution date.

1. drainReprocessQueueReprocessQueue processing queue, that is received by repocess task. When pendingTasks task queue exceeds the threshold value, the retry task discarded.
2. drainAcceptorQueueAcceptorQueue processing queue, that is received by the task process. pendingTasks oldest tasks directly discarded, will add a new task to the queue.

private void drainReprocessQueue() {
    long now = System.currentTimeMillis();
    // 1. 只要 pendingTasks 没有超过阈值，maxBufferSize=10000
    //    就将重试的任务添加到 pendingTasks 中 
    while (!reprocessQueue.isEmpty() && !isFull()) {
        TaskHolder<ID, T> taskHolder = reprocessQueue.pollLast();
        ID id = taskHolder.getId();
        if (taskHolder.getExpiryTime() <= now) {
            expiredTasks++;
        } else if (pendingTasks.containsKey(id)) {
            overriddenTasks++;
        } else {
            pendingTasks.put(id, taskHolder);
            processingOrder.addFirst(id);
        }
    }
    // 2. reprocessQueue 队列中剩余的任务全部丢弃。
    if (isFull()) {
        queueOverflows += reprocessQueue.size();
        reprocessQueue.clear();
    }
}

drainReprocessQueue method when too many tasks directly discarded retry tasks, drainAcceptorQueue is different, discard the oldest task, task execution date. The purpose is to ensure that new tasks can certainly perform, and discard the old tasks according to the actual situation.

private void drainAcceptorQueue() {
    while (!acceptorQueue.isEmpty()) {
        appendTaskHolder(acceptorQueue.poll());
    }
}
private void appendTaskHolder(TaskHolder<ID, T> taskHolder) {
    // 1. 如果任务超出阈值，丢弃最老的任务
    if (isFull()) {
        pendingTasks.remove(processingOrder.poll());
        queueOverflows++;
    }
    // 2. 将最新的任务添加到队列中
    TaskHolder<ID, T> previousTask = pendingTasks.put(taskHolder.getId(), taskHolder);
    if (previousTask == null) {
        processingOrder.add(taskHolder.getId());
    } else {
        overriddenTasks++;
    }
}

Summary: Again, if the implementation of the principle of pendingTasks exceeds the threshold are:

1. Discard the oldest tasks and retry tasks, task execution date.

2. With taskId task execution only the latest task .

   appendTaskHolder taskId task overrides the same name, taskId generation PeerEurekaNode is generated when the received message broadcasting task, generating principle is: requestType（任务类型）+ appName（应用名称）+ id（实例id）. Task types include: register, cancel, heartbeat, statusUpdate , deleteStatusOverride.

   // PeerEurekaNode.process 是会生成 taskId  
   private static String taskId(String requestType, InstanceInfo info) {
       return taskId(requestType, info.getAppName(), info.getId());
   }
   private static String taskId(String requestType, String appName, String id) {
       return requestType + '#' + appName + '/' + id;
   }
   

2.2.3 assignBatchWork

The task to move from the batchWorkQueue, requestWorkItem batchWorkQueue processed directly obtained from pendingTasks.

// pendingTasks -> batchWorkQueue
void assignBatchWork() {
    // 1. pendingTasks 为空则 false，pendingTasks 队列满了肯定为 true。
    //    任务的延迟时间不超过 maxBatchingDelay=500ms
    if (hasEnoughTasksForNextBatch()) {
        if (batchWorkRequests.tryAcquire(1)) {
            long now = System.currentTimeMillis();
            // 2. 批处理任务最大为 maxBatchingSize=250
            int len = Math.min(maxBatchingSize, processingOrder.size());
            List<TaskHolder<ID, T>> holders = new ArrayList<>(len);
            while (holders.size() < len && !processingOrder.isEmpty()) {
                ID id = processingOrder.poll();
                TaskHolder<ID, T> holder = pendingTasks.remove(id);
                // 3. 任务过期，直接丢弃
                if (holder.getExpiryTime() > now) {
                    holders.add(holder);
                } else {
                    expiredTasks++;
                }
            }
            // 4. 添加到 batchWorkQueue 队列中
            if (holders.isEmpty()) {
                batchWorkRequests.release();
            } else {
                batchSizeMetric.record(holders.size(), TimeUnit.MILLISECONDS);
                batchWorkQueue.add(holders);
            }
        }
    }
}

Summary: Conditions assignBatchWork do: First, pendingTasks task overload, and immediate implementation; the second is the task delay time is greater than maxBatchingDelay = 500ms. The purpose is to perform frequency control tasks: Task too much or too long delayed execution immediately.

private boolean hasEnoughTasksForNextBatch() {
    if (processingOrder.isEmpty()) {
        return false;
    }
    // 队列中任务大多立即执行 maxBufferSize=10000
    if (pendingTasks.size() >= maxBufferSize) {
        return true;
    }
	// 任务延迟时间太长立即执行 maxBatchingDelay=500ms
    TaskHolder<ID, T> nextHolder = pendingTasks.get(processingOrder.peek());
    long delay = System.currentTimeMillis() - nextHolder.getSubmitTimestamp();
    return delay >= maxBatchingDelay;
}

3. Analysis of Key Technologies

3.1 Consumer speed control

(1) server is busy or network anomalies

When the server is busy or network IO abnormal when you need to wait before sending the request. by default:

• congestionRetryDelayMs: at least 100ms when the server is busy
• networkFailureRetryMs: network IO exception least 1000ms

public void run() {
    ...
    if (scheduleTime < now) {
        scheduleTime = now + trafficShaper.transmissionDelay();
    }
    if (scheduleTime <= now) {
        assignBatchWork();
        assignSingleItemWork();
    }
}

Summary: If the server is busy (503) or network IO exception, at least to wait a certain period of time, send the request again. TrafficShaper flow is meant the entire line, i.e. the control frequency of the transmission request.

long transmissionDelay() {
    // 没有任务异常，不能等待
    if (lastCongestionError == -1 && lastNetworkFailure == -1) {
        return 0;
    }
	// 出现对方服务器忙，至少 congestionRetryDelayMs=100ms
    long now = System.currentTimeMillis();
    if (lastCongestionError != -1) {
        long congestionDelay = now - lastCongestionError;
        if (congestionDelay >= 0 && congestionDelay < congestionRetryDelayMs) {
            return congestionRetryDelayMs - congestionDelay;
        }
        lastCongestionError = -1;
    }

	// 出现网IO异常，至少 networkFailureRetryMs=1000ms
    if (lastNetworkFailure != -1) {
        long failureDelay = now - lastNetworkFailure;
        if (failureDelay >= 0 && failureDelay < networkFailureRetryMs) {
            return networkFailureRetryMs - failureDelay;
        }
        lastNetworkFailure = -1;
    }
    return 0;
}

(2) limit the maximum number of tasks

If the message broadcast task is beyond the threshold, discarding the principle is: First, discard the oldest tasks and retry tasks, task execution date. The second is to perform the same task taskId only the latest task . This is described in detail in section 2.2.2, the default maxBufferSize = 10000.

(3) batch job delay

In PeerEurekaNode receive broadcast task, the generation TaskHolder, will generate job submission time, if the task is delayed hurry exceeded maxBatchingDelay is executed immediately. This time is controlled in hasEnoughTasksForNextBatch process. Default maxBatchingDelay = 500ms.

3.2 Semaphore batchWorkRequests effect analysis

In the process of the task will use this signal lock Semaphore, what it does is it?

private final Semaphore batchWorkRequests = new Semaphore(0);
private final BlockingQueue<List<TaskHolder<ID, T>>> batchWorkQueue = new LinkedBlockingQueue<>();

// AcceptorRunner 分配任务
void assignBatchWork() {
    if (hasEnoughTasksForNextBatch()) {
        if (batchWorkRequests.tryAcquire(1)) {
            List<TaskHolder<ID, T>> holders = new ArrayList<>(len);
           	...
            if (holders.isEmpty()) {
                // 如果没有分配任务，下一次可继续分配任务
                batchWorkRequests.release();
            } else {
                // 如果已经分配任务，则必须等到消费才消费才开始重新分配任务
                // 如果任务一直没有被消费，则 AcceptorRunner 轮询时会丢弃老的任务
                batchWorkQueue.add(holders);
            }
        }
    }
}
// WorkerRunable 获取任务
BlockingQueue<List<TaskHolder<ID, T>>> requestWorkItems() {
    batchWorkRequests.release();
    return batchWorkQueue;
}

Summary: task allocation when AcceptorRunner thread polling, if no lock Semaphore, that task has not been consumed, when too much pendingTasks task, according to discard the old new tasks for processing principles. If there WorkerRunable thread consumption will release the lock, re-tasking.

---

The intentions of recording a little bit every day. Perhaps the content is not important, but the habit is very important!