Hongmeng kernel source code analysis (dispatch queue)

Dispatch Queue

Note: Based on core open source obscurity analysis, the official source [ kernel_liteos_a ] official documents [ docs ] Reference Document [ Huawei LiteOS ]
author: Hong Meng core enthusiasts, will continue to study obscurity kernel, update blog, so stay tuned. The content only represents personal views, errors are welcome, everyone is welcome to correct and improve. All articles in this series enter to view the  source code analysis of Hongmeng system (general catalog)

---

table of Contents

Dispatch Queue

Why talk about the dispatch queue alone?

Functions involved

 Bitmap scheduler

Task ready queue mechanism

Several commonly used functions

Can the priority of threads in the same process be different?

How to understand thread group?

All series of articles enter Hongmeng system source code analysis (general catalog) view

---

---

Why talk about the dispatch queue alone?

There are two source files in the Hongmeng kernel code about queues, one is the queue for scheduling, and the other is the IPC queue for inter-thread communication. 

This article describes the scheduling queue in detail , see the code: kernel_liteos_a/kernel/base/sched/sched_sq/los_priqueue.c

A follow-up blog post about the IPC queue describes that the data structure of these two queues are implemented using a two-way circular linked list. LOS_DL_LIST is really important and the key to understanding the Hongmeng kernel. It is not an exaggeration to say that it is the most important code. , The source code appears in the sched_sq module, indicating that it is used for task scheduling. The sched_sq module has only two files, and the other los_sched.c is the scheduling code.

Why do you have to come up with an article to talk about the dispatch queue? Because the queue is the pre-storage for scheduling, if you compare the task to grain, the queue is the warehouse where the grain is loaded. The scheduling algorithm is the person who transports the grain. Of course, the grain and the person who carry it are very important, but the warehouse is also very important. They are scheduling. Provides the source of the task, where the task comes from, and where it goes are all completed by the logic carried by these queues.

Functions involved

Function classification	Interface name	description
Create queue	OsPriQueueInit	32 ready queues created
Get the highest priority queue	OsPriQueueTop	Check the highest priority task
Enter the queue from the head	OsPriQueueEnqueueHead	Insert a ready queue from the head
Enter the queue from the tail	OsPriQueueEnqueue	The default is to insert a ready queue from the tail
Dequeue	OsPriQueueDequeue	Remove from the highest priority ready queue
	OsPriQueueProcessDequeue	Remove from the process queue
	OsPriQueueProcessSize	Use the process to check the number of elements in the queue
	OsPriQueueSize	Use task to check the number of elements in the queue
	OsTaskPriQueueTop	Check the highest priority task
	OsDequeEmptySchedMap	Process dequeue
	OsGetTopTask	Get the task selected by scheduling

 Bitmap scheduler

//*kfy 0x80000000U = 10000000000000000000000000000000(32位,1是用于移位的，设计之精妙，点赞) 
#define PRIQUEUE_PRIOR0_BIT   0x80000000U 

#ifndef CLZ
#define CLZ(value)                                  (__clz(value)) //汇编指令
#endif

LITE_OS_SEC_BSS LOS_DL_LIST *g_priQueueList = NULL; //所有的队列 原始指针
LITE_OS_SEC_BSS UINT32 g_priQueueBitmap; // 位图调度
// priority = CLZ(bitmap); // 获取最高优先级任务队列 调度位

The entire los_priqueue.c has only two all variables, one is LOS_DL_LIST *g_priQueueList is the head pointer of 32 process ready queues, and another UINT32 g_priQueueBitmap will be mentioned in the ready queue. It is estimated that many people will be unfamiliar. It is a 32-bit variable called Bitmap scheduler. How to understand it?

The scheduling of the Hongmeng system is preemptive. Tasks are divided into 32 priority levels. How to quickly know which queue is empty and which task needs an identification, and it must be implemented extremely efficiently? The answer is: bitmap scheduler.

Simply put, it is a variable bit to mark whether there is a task in the corresponding queue. Under bitmap scheduling, the smaller the task priority value, the higher the priority. Whenever scheduling is required, from the lowest to the highest The bit finds the location of the first bit set to 1, which is the current highest priority, and then obtains the corresponding task control block from the corresponding priority ready queue. The implementation complexity of the entire scheduler is O(1), that is, no matter The scheduling time of tasks is fixed.

Task ready queue mechanism

The CPU execution speed is very fast, and its computing speed and memory read and write speed are orders of magnitude difference, and it is exponential with hard disk read and write speed. The default time slice of the Hongmeng kernel is 10ms. The resource is very precious. It constantly switches back and forth among many tasks, so you must never let the CPU wait for the task. The CPU is like the big boss of the company. Many departments below wait for the boss to approve and eat. Only everyone is waiting for the leader, there is no reason why the leader is waiting for you, so the work must be prepared in advance, and the priority of each department is different, so there is a task queue corresponding to the priority, and all the leaders can directly handle. Task, don't let it in if you are not ready.
This is the mechanism of the task ready queue. There are a total of 32 task ready queues, because the thread priority is 32 by default, and each queue puts tasks of the same priority.

What does the queue initialization do? See the code in detail

#define OS_PRIORITY_QUEUE_NUM 32

UINT32 OsPriQueueInit(VOID)
{
    UINT32 priority;

    /* system resident resource */
    g_priQueueList = (LOS_DL_LIST *)LOS_MemAlloc(m_aucSysMem0, (OS_PRIORITY_QUEUE_NUM * sizeof(LOS_DL_LIST)));
    if (g_priQueueList == NULL) {
        return LOS_NOK;
    }

    for (priority = 0; priority < OS_PRIORITY_QUEUE_NUM; ++priority) {
        LOS_ListInit(&g_priQueueList[priority]);
    }
    return LOS_OK;
}

Because TASK has 32 priority levels, the kernel creates 32 two-way circular linked lists at one time during initialization. Each priority level has a queue to record the position of the tasks in the ready state. g_priQueueList allocates a continuous memory block for storage There are 32 LOS_DL_LISTs, let’s look at the LOS_DL_LIST structure again, because it is so important! The simpler the more flexible

typedef struct LOS_DL_LIST {
    struct LOS_DL_LIST *pstPrev; /**< Current node's pointer to the previous node */
    struct LOS_DL_LIST *pstNext; /**< Current node's pointer to the next node */
} LOS_DL_LIST;

Several commonly used functions

Look at the source code of entering and leaving the team, pay attention to the changes of bitmap!

From the code, we can know that LOS_ListTailInsert(&priQueueList[priority], priqueueItem); is called. Note that it is inserted from the end of the circular linked list, that is, TASKs of the same priority are ranked in the last execution, as long as each time is from the end Insertion forms a queue for sequential execution. The design of Hongmeng's kernel is very clever, with very little code and extremely efficient implementation of the queue function.

VOID OsPriQueueEnqueue(LOS_DL_LIST *priQueueList, UINT32 *bitMap, LOS_DL_LIST *priqueueItem, UINT32 priority)
{
    /*
     * Task control blocks are inited as zero. And when task is deleted,
     * and at the same time would be deleted from priority queue or
     * other lists, task pend node will restored as zero.
     */
    LOS_ASSERT(priqueueItem->pstNext == NULL);

    if (LOS_ListEmpty(&priQueueList[priority])) {
        *bitMap |= PRIQUEUE_PRIOR0_BIT >> priority;//对应优先级位 置1
    }

    LOS_ListTailInsert(&priQueueList[priority], priqueueItem);
}

VOID OsPriQueueEnqueueHead(LOS_DL_LIST *priQueueList, UINT32 *bitMap, LOS_DL_LIST *priqueueItem, UINT32 priority)
{
    /*
     * Task control blocks are inited as zero. And when task is deleted,
     * and at the same time would be deleted from priority queue or
     * other lists, task pend node will restored as zero.
     */
    LOS_ASSERT(priqueueItem->pstNext == NULL);

    if (LOS_ListEmpty(&priQueueList[priority])) {
        *bitMap |= PRIQUEUE_PRIOR0_BIT >> priority;//对应优先级位 置1
    }

    LOS_ListHeadInsert(&priQueueList[priority], priqueueItem);
}

VOID OsPriQueueDequeue(LOS_DL_LIST *priQueueList, UINT32 *bitMap, LOS_DL_LIST *priqueueItem)
{
    LosTaskCB *task = NULL;
    LOS_ListDelete(priqueueItem);

    task = LOS_DL_LIST_ENTRY(priqueueItem, LosTaskCB, pendList);
    if (LOS_ListEmpty(&priQueueList[task->priority])) {
        *bitMap &= ~(PRIQUEUE_PRIOR0_BIT >> task->priority);//队列空了，对应优先级位 置0
    }
}

Can the priority of threads in the same process be different?

Please think about this first.

Process and thread have a one-to-many parent-child relationship. The unit of kernel scheduling is task (thread). In Hongmeng kernel, task and thread are one thing, but different identities. A process can have multiple threads, and the threads have their own independent state, how should the process state be defined? For example: ProcessA has TaskA (blocking state) and TaskB (ready state). Does ProcessA belong to the blocking state or the ready state?

First look at the description of the official document and then look at the source code.

Process state transition description:

• Init→Ready：

  When a process is created or fork, it enters the Init state after getting the process control block and is in the process initialization stage. When the process initialization is completed, the process is inserted into the scheduling queue, and the process enters the ready state.

• Ready→Running：

  After the process is created, it enters the ready state. When a process switch occurs, the process with the highest priority in the ready list is executed and enters the running state. If no other threads in the process are in the ready state at this time, the process is deleted from the ready list and only in the running state; if there are other threads in the process in the ready state at this time, the process is still in the ready queue. The ready state and the running state of the process coexist.

• Running→Pend：

  When all the threads in the process are in the blocking state, when the last thread of the process turns into the blocking state, it enters the blocking state synchronously, and then the process switch occurs.

• Pend→Ready / Pend→Running：

  When any thread in the blocked process returns to the ready state, the process is added to the ready queue and synchronously changes to the ready state. If a process switch occurs at this time, the process state changes from the ready state to the running state.

• Ready→Pend：

  When the last ready state thread in the process is in the blocking state, the process is deleted from the ready list, and the process changes from the ready state to the blocking state.

• Running→Ready：

  There are two situations in which the process changes from the running state to the ready state:

  1. After a process with a higher priority is created or restored, process scheduling occurs. At this moment, the highest priority process in the ready list becomes the running state, and the originally running process changes from the running state to the ready state.
  2. If the scheduling strategy of the process is SCHED_RR, and another process with the same priority is in the ready state, after the time slice of the process is exhausted, the process turns from the running state to the ready state, and another process with the same priority changes from the ready state The state changes to the running state.

• Running→Zombies：

  When the main thread or all threads of the process are finished, the process turns from the running state to the zombie state, waiting for the parent process to reclaim resources.

 Pay attention to the red part above, a process can coexist in two states!

    UINT16               processStatus;                /**< [15:4] process Status; [3:0] The number of threads currently
                                                            running in the process */

    processCB->processStatus &= ~(status | OS_PROCESS_STATUS_PEND);//取反后的与位运算
    processCB->processStatus |= OS_PROCESS_STATUS_READY;//或位运算

One variable has two states, how to do it? The answer is still bitwise. Remember the bitmap scheduling g_priQueueBitmap above, which has 32 states. In fact, this is very common in the kernel source code of any system, similarly, there are shift left <<, shift right >> etc.

Continue to talk about the relationship between processes and threads, must the priority of threads be the same as processes? Can they be different? The answer is: it can be different, otherwise, how can there be a function to set task priority.

How to understand thread group?

What really makes the CPU work is the thread. The process is just a container that holds the thread. The thread has task stack space and runs independently in the kernel space. The process has only user space. We will talk about it in the subsequent memory chapter. We will not expand it here. The process structure LosProcessCB has such a definition. Just look at the name, it's related to scheduling.

    UINT32               threadScheduleMap;            /**< The scheduling bitmap table for the thread group of the
                                                            process */
    LOS_DL_LIST          threadPriQueueList[OS_PRIORITY_QUEUE_NUM]; /**< The process's thread group schedules the
                                                                         priority hash table */

At first glance, there are 32 queues in the process structure. In fact, this is the ready state queue of the thread. threadScheduleMap is the process's own bitmap scheduler. See the source code of the process enqueue and dequeue. The scheduling process is to first find the highest priority process in the process ready queue, and then go to the process to find the highest priority thread for scheduling. Specifically, I think the kernel's most beautiful function OsGetTopTask, can appreciate its beauty, and understand how the ready queue is managed. 

LITE_OS_SEC_TEXT_MINOR LosTaskCB *OsGetTopTask(VOID)
{
    UINT32 priority, processPriority;
    UINT32 bitmap;
    UINT32 processBitmap;
    LosTaskCB *newTask = NULL;
#if (LOSCFG_KERNEL_SMP == YES)
    UINT32 cpuid = ArchCurrCpuid();
#endif
    LosProcessCB *processCB = NULL;
    processBitmap = g_priQueueBitmap;
    while (processBitmap) {
        processPriority = CLZ(processBitmap);
        LOS_DL_LIST_FOR_EACH_ENTRY(processCB, &g_priQueueList[processPriority], LosProcessCB, pendList) {
            bitmap = processCB->threadScheduleMap;
            while (bitmap) {
                priority = CLZ(bitmap);
                LOS_DL_LIST_FOR_EACH_ENTRY(newTask, &processCB->threadPriQueueList[priority], LosTaskCB, pendList) {
#if (LOSCFG_KERNEL_SMP == YES)
                    if (newTask->cpuAffiMask & (1U << cpuid)) {
#endif
                        newTask->taskStatus &= ~OS_TASK_STATUS_READY;
                        OsPriQueueDequeue(processCB->threadPriQueueList,
                                          &processCB->threadScheduleMap,
                                          &newTask->pendList);
                        OsDequeEmptySchedMap(processCB);
                        goto OUT;
#if (LOSCFG_KERNEL_SMP == YES)
                    }
#endif
                }
                bitmap &= ~(1U << (OS_PRIORITY_QUEUE_NUM - priority - 1));
            }
        }
        processBitmap &= ~(1U << (OS_PRIORITY_QUEUE_NUM - processPriority - 1));
    }

OUT:
    return newTask;
}

Summary: If you understand OsGetTopTask, please leave a message in the comment area.

All series of articles enter Hongmeng system source code analysis (general catalog) view