1. Operating system overview

1.1 Definition and goals of the operating system

Definition: The operating system is the hardware and software that controls and manages the computer system, and the system software that allocates and schedules resources .

Goals: Convenience, effectiveness (enhance the utilization rate of system resources, increase the throughput of the system) , expandability, and openness.

1.2 Basic functions of the operating system

Unified management of computer resources : processor resources, IO device resources, memory resources, file resources;
The abstraction of computer resources is realized : the IO device management software provides a read-write interface, and the file management software provides an operation file interface;
Provides an interface between the user and the computer : GUI (Graphical User Interface), command form, and system call form.

1.3 Characteristics of the operating system

The most basic features are mutual existence conditions: concurrency and sharing;

(1) Parallelism : means that two or more events can occur at the same time , multi-core CPUs can achieve parallelism , and only one program is running on a CPU at the same time;

(2) Concurrency : It means that two or more events can occur at the same time interval . It seems to the user that each program is running, but in fact each program is executed alternately .

(3) Sharing : The resources in the operating system can be shared by multiple concurrent programs. This form is called resource sharing .

Mutually exclusive sharing : When resources are occupied by a program, other programs that want to use them can only wait.
Simultaneous access : A resource is accessed concurrently by multiple programs.

The premise of virtual and asynchronous features is concurrency .

(4) Virtuality : It is manifested as transforming a physical entity into several logical entities.

Time-division multiplexing technology : resources are multiplexed in time, different programs are used concurrently, and multi-programs use computer hardware resources in time-sharing to improve resource utilization.
Space-division multiplexing technology : used to realize virtual disks (physical disks are virtualized as logical disks, C disks and D disks on computers, etc.), virtual memory (logically expand the storage capacity of programs), etc., to improve resource utilization. Improve programming efficiency.

(5) Asynchronicity : In a multi-programming environment, multiple processes are allowed to execute concurrently, but due to resource constraints and other factors, the execution of the process runs in a "stop and go" manner, and the execution of each process (Run, Pause, Speed, Completion) are also unknown .

1.4 Interrupt handling of the operating system

The role of the interrupt mechanism: to allow users to interact in a multi-channel batch processing system;

Interrupt generation :

When an interrupt occurs, the CPU immediately switches to the management state to carry out management work; (the management state is also called the privileged state, system state or core state, which is the state the machine is in when the program managed by the operating system is executed.)
After an interruption occurs, the currently running process will be suspended, and the operating system kernel will handle the interruption;
For different interrupt signals, different processing will be carried out.

Classification of interruptions :

Internal interrupt (also known as "exception", "exception", "trapped") ------- signal source: inside the CPU , related to the currently executing instruction;
External interrupt (interrupt)----------Signal source: Outside the CPU , it has nothing to do with the current execution instruction.

Processing of external interrupts :

After each instruction is executed, the CPU needs to check whether there is an external interrupt signal;
If an external interrupt signal is detected, it is necessary to protect the CPU environment of the interrupted process (such as the program status word PSW, program counter PC, various general-purpose registers) and store them in the PCB (process control block);
Transfer to the corresponding interrupt handler according to the interrupt signal type;
Restore the CPU environment of the original process and exit the interrupt, and return to the original process to continue execution.

2. Process management

2.1 Process entity of process management

Why process is needed :

Process is the basic unit of system resource allocation and scheduling ;
The process, as the carrier of the independent operation of the program, guarantees the normal execution of the program;
The existence of the process greatly improves the utilization rate of operating system resources. +

Process control block (PCB) : a general data structure used to describe and control the operation of the process, record the current state of the process and all information about the operation of the control process, and is the unique identification of the existence of the process .

Process (Process) and thread (Thread) :

Threads : Operating System Conduct**The smallest unit of running scheduling**。
Process : System in progress**The basic unit of resource allocation and scheduling**。

difference and connection：

A process can have one or more threads ;
A thread is included in a process and is the unit that actually runs work in the process ;
Process threads share process resources ;
A process can have multiple threads running concurrently, and each thread performs a different task .

2.2 Five-state model of process management

Ready state : Other resources (process control block, memory, stack space, heap space, etc.) are all ready, only the state of the CPU.
Execution state : The process got the CPU and its program is executing.
Blocked state : The state of the process giving up the CPU for some reason, and the blocked process is placed in the form of a queue.
Creation status : When the process is created, the PCB is owned but other resources are not yet ready.
Termination status : The process ends and the system cleans up or returns the PCB status.

2.3 Process synchronization of process management

Producer-consumer problem : There are a group of producer processes producing products and providing these products to the consumer process for consumption. The producer process and the consumer process can be executed concurrently. There is a set of n between them. The buffer pool of the buffer, the producer process needs to put the produced product in the buffer (+1 operation), and the consumer process can take the product consumption from the buffer (-1 operation).

A problem : when the two are executed concurrently, an error may occur, causing the expected result to be inconsistent with the real result: when the producer +1 and consumer -1 operations are executed, the value of the buffer changes from 10 to 11 .

Dining with philosophers : There are 5 philosophers whose way of life is to think and eat alternately. The philosophers share a round table and sit on 5 chairs. There are 5 bowls and 5 chopsticks on the round table. Usually philosophers only think, and when they are hungry, they try to take the left and right chopsticks that are close to them. Only when both chopsticks are taken can they eat, otherwise they wait, and after eating, put down the left and right chopsticks to think.

This can lead to the following problems, chopsticks are equivalent to critical resources:

Critical resources refer to some shared resources that cannot be accessed by multiple threads at the same time even though they are shared resources. When a process is using a critical resource, other processes must wait for the occupying process to release the shared resource according to the synchronization mechanism of the operating system before re-competing to use the shared resource.

The role of process synchronization : to coordinate the use order of competing resources among multiple processes, so that multiple concurrently executing processes can effectively use resources and cooperate with each other .

Four principles of inter-process synchronization :

Idle give-in : the resource is not occupied and is allowed to be used;
Wait while busy : the resource is occupied, and the requesting process waits;
Limited waiting : guarantee limited waiting time to use resources;
Give up the right to wait : When waiting, the process needs to give up the CPU.

2.3.1Method of process synchronization (important)

1. Use the fork system call to create a process : use the fork system call without parameters, fork will return twice, return the child process id and 0 respectively, the parent process returns the child process id, and the child process returns 0.

The fork system call is used to create a process;
The initialization state of the process created by fork is the same as that of the parent process;
The system will allocate new resources for the fork process

2. Shared memory : To some extent, multiple processes share physical memory, but due to the process management of the operating system, the memory space between processes is independent, so the process cannot access the memory space outside the process space by default. of.

Shared storage allows unrelated processes to access the same piece of physical memory;
Shared memory is the fastest way to share and pass data between two processes ;
Shared memory does not provide a synchronization mechanism , and other mechanisms need to be used to manage access;

3. Unix Domain Sockets

Domain socket is an advanced inter-process communication method that can be used for inter-process communication on the same machine.

Socket (socket): A term used in network communication.

The domain socket provided by the Unix system provides similar functions to the network socket, such as Nfinx, uWSGI, etc.

The process of using Unix domain sockets on the server and the client respectively:

2.3.2 Method of thread synchronization (important)

The method of thread synchronization :

Mutex : Mutex is the simplest method of thread synchronization, also known as mutex, a variable in one of two states: unlocked and locked . The two states can guarantee the serialization of resource access. Atomicity: Refers to the characteristic that a series of operations cannot be interrupted , either all of them are executed or none of them are executed.
Spin lock : A spin lock is a multi-thread synchronization variable. The thread using the spin lock will repeatedly check whether the lock variable is available . The spin lock will not give up the CPU . It is a busy waiting state , that is, an infinite loop waiting The lock is released , and the efficiency of the spin lock is much higher than that of the mutex . Features: Avoids the overhead of process or thread context switching , but is not suitable for use on single-core CPUs .
Read-write lock : It is a special spin lock that allows multiple read operations to access resources at the same time to improve read performance , but it is mutually exclusive for write operations , ie **Improve the operation efficiency of more reads and less writes**Significantly.
Condition variable : It is a relatively complex thread synchronization method. The condition variable allows the thread to sleep until a certain condition is met, when**To meet the conditions, the thread can be signaled to wake up**.

2.3.3 Comparison of thread synchronization methods (important)

2.4 Linux process management

Process type :

Foreground process : has a terminal and can interact with users;
Background process : does not occupy the terminal, basically does not interact with the user, and has a lower priority than the foreground process (the command that needs to be executed ends with the "&" symbol);
Daemon process : A special background process that starts when the system boots and runs until the system is shut down (process names ending with "d" are generally daemon processes), such as crond, sshd, httpd, mysqld...

Process flags :

Process ID : non-negative integer, the unique mark of the process, each process has a different ID;
The status flag of the process : R indicates that the process is running, S indicates that the process is sleeping...

Related commands to operate Linux processes :

ps command : list the current process, combined with -aux can print the detailed information of the process (ps -aux);
top command : view the status of all processes;
kill command : Send a signal to the process.

3. Job management

3.1 Process scheduling of job management

Definition: Refers to the computer's decision to determine which ready process can obtain CPU usage rights .

When is process scheduling needed ?

Actively give up: the process terminates normally; terminates due to an exception during operation; actively blocks (such as waiting for I/O);
Passive abandonment: the time slice allocated to the process is exhausted; a higher priority process enters the ready queue; there are more urgent things to deal with (such as I/O interruption);

Process scheduling method :

Non-preemptive scheduling : only the currently running process can actively give up the CPU ;

Once the processor is assigned to a process, let the process continue to use it;
The scheduler does not preempt the processor being used for any reason;
The scheduler does not preempt the processor being used for any reason;

Preemptive scheduling : The operating system can deprive the CPU usage right of the current process .

Allow the scheduler to suspend the currently running process with a certain policy;
Save the context information of the old process and assign the processor to the new process;

Three mechanisms for process scheduling :

The queuing mechanism of the ready queue : In order to improve the efficiency of process scheduling, the ready processes are queued in a certain way, so that the scheduler can find the ready process as soon as possible.

The delegation mechanism for selecting the running process : the scheduler selects the ready process with a certain strategy and allocates CPU resources to it.

Context switching mechanism for old and new processes : save the context information of the current process and load the running context of the delegated execution process.

Process scheduling algorithm :

First-come-first-serve algorithm : Execute in the order in which they are in the ready queue .
Short process priority scheduling algorithm : Prioritize the process with the shortest estimated running time in the ready queue , which is not conducive to the execution of long job processes.
High-priority priority scheduling algorithm : Processes have priority, and priority is given to processes with high weights , which can make urgent tasks prioritized.
Time slice round-robin scheduling algorithm : according to the principle of FIFO, the ready processes are arranged, each time the process to be executed is taken out from the head of the queue, and a time slice is allocated for execution . This is a relatively fair scheduling algorithm, but it cannot guarantee that it will respond to users.

3.2 Deadlock of job management

3.2.1 The difference between process deadlock, starvation, and infinite loop:

Deadlock : During the execution of two or more processes , due to competition for resources or a blocking phenomenon caused by mutual communication , if there is no external force, they will not be able to advance. Processes that are forever waiting for each other are called deadlocked processes.

Hunger : process stalled due to chronic lack of access to resources ;

Infinite loop : code logic BUG.

The occurrence of deadlock : competition for resources (the number of shared resources does not meet the needs of each process), improper process scheduling sequence, when the scheduling sequence is A->B->C->D, deadlock will occur, but it is changed to A->D ->B->C will not be generated.

Four necessary conditions for deadlock：

Mutual exclusion conditions : Deadlocks must occur only when resources are used in mutual exclusion ;
Request holding conditions : the process holds at least one resource , and makes a new resource request, the new resource is occupied, the request is blocked , and the blocked process does not release the resource it holds;
Non-alienable conditions : The resources obtained by the process cannot be deprived (including OS) before they are fully used, and can only be released by the process itself ;
Loop waiting condition : When a deadlock occurs, there must be a process-resource ring chain. Loop waiting does not necessarily cause deadlock, but deadlock must have cyclic waiting.

deadlock handling strategy：

1. Ways to prevent deadlock : destroy one or more of the four necessary conditions.

Destruction of mutual exclusion conditions : Transform critical resources into shared resources (Spooling pooling technology); (feasibility is not high, and mutual exclusion conditions cannot be destroyed in many cases)
Destruction of request retention conditions : before the system stipulates that the process runs, all required resources are requested at one time ; (low resource utilization may lead to starvation of other threads)
Destruction of the inalienable condition : When a process requests new resources and cannot be satisfied, the occupied resources must be released ; (the implementation is complex, the deprivation of resources may cause some work to fail, and repeated applications and releases cause additional system overhead)
Breaking the loop waiting condition : the available resources are sorted linearly , and the application must be incrementally applied according to the need; (the actual resource usage sequence and the number sequence of the process are different, which will lead to waste of resources)

2. Banker's Algorithm : Check whether the remaining resources can meet the maximum demand of a certain process; if so, add the process to the security sequence, wait for the process to be allowed to complete, and recycle all resources; repeat 1, 2 until there are no threads waiting resource;

3. Deadlock detection and release : deadlock detection algorithm, resource deprivation method, undo process method (termination process method), process rollback method;

4. Storage Management

Storage management is to ensure that the computer has enough memory to process data; to ensure that programs can obtain a portion of memory usage from available memory; to ensure that programs can return used memory for use by other programs.

4.1 Memory allocation and recycling of storage management

The process of memory allocation: single continuous allocation (obsolete), fixed partition allocation, dynamic partition allocation (dynamically allocate memory according to actual needs).
Dynamic partition allocation algorithm :

First adaptation algorithm : When allocating memory, search for suitable memory areas sequentially from the beginning. If there is no suitable memory area, the allocation will fail, starting from the head each time, so that the head address space is continuously divided;
Best Adaptive Algorithm : The linked list of free areas is required to be sorted according to capacity, and traversed to find the best suitable free area (leaving more and more internal fragments).
Fast Adaptive Algorithm : Multiple free area linked lists are required, and each free area linked list stores a free area of a capacity.

The process of memory recovery :

The recovery area is below the free area : no need to create a new free list node; only need to increase the capacity of the free area 1;
The recycling area is above the free area : merge the recycling area with the free area; the new free area uses the address of the recycling area;
The recovery area is in the middle of the free area : merge the free area 1, the free area 2 and the recovery area; the new free area uses the address of the free area 1;
Only the remaining recovery area : create a new free node for the recovery area; insert it into the corresponding free area linked list;

4.2 Segment page storage management of storage management

Paging storage management : Divide the process logical space into pages of several sizes , and correspondingly divide the physical memory space into physical blocks of page size, and load the process space into scattered physical blocks in the physical memory in units of pages.

The page size should be moderate, if it is too large, it is difficult to allocate, if it is too small, there are too many memory fragments; the page size is usually 512B~8K ;

Modern computer systems can support a very large logical address space (2 ^32~2 64). A paging system with a 32-bit logical address space stipulates that the page size is 4KB, and the page table entries in the page table of each process can be Up to 1M (2 20), if each page table entry occupies 1Byte, so each process will occupy 1MB of memory space only for the page table.

Segment storage management : Divide the process logic space into several segments (unequal division) , and the length of the segment is determined by the length of the continuous logic.

Compared with paging and segment storage management :

Both segment storage and page storage manage the logical space of the process discretely ;
A page is a physical unit, and a segment is a logical unit ;
Paging is to make reasonable use of space, and segmentation is to meet user requirements. The page size is fixed by hardware , and the segment length can be changed dynamically ;
The page table information is one-dimensional, and the segment table information is two-dimensional ;

Segmented page storage management : The logical space is now divided into several segments according to the segmented management, and then the memory space is divided into several pages according to the paged management. Paging can effectively improve memory utilization , and segmentation can better meet user needs .

4.3 Virtual memory for storage management

Overview of virtual memory : It is the key technology of operating system memory management, which makes multi-program operation and large-scale program operation a reality, divides the memory used by the program, and places some temporarily unused memory in auxiliary storage, which is actually an expansion of physical memory . .
Locality principle : When the CPU accesses the memory, whether it is accessing instructions or accessing data , the accessed storage units tend to gather in a smaller continuous area .
Virtual memory replacement algorithm : first in first out (FIFO), least frequently used (LFU), least recently used (LRU)

Features of virtual memory :

Multiplicity : It is not necessary to load all the jobs into the memory at one time when the job is running, but allows it to be divided into multiple times and loaded into the memory;
Swapability : It does not need to be resident in the memory when the job is running, but allows the job to be swapped in and swapped out during the running of the job;
Virtuality : Logically expands the memory capacity, so that the memory used by the user is much larger than the actual capacity;

4.4 Linux storage management

Buddy memory management algorithm : a classic memory management algorithm, in order to solve the problem of out-of-memory fragmentation , the algorithm has extremely high efficiency based on the advantages of computer processing binary.
Linux swap space : Swap space (Swap) is a partition of the disk. When the Linux memory is full, some memory will be swapped to the Swap space. The Swap space is configured when the system is initialized.
Comparison of Swap space and virtual memory :

5. Document Management

5.1 File management of the operating system

The logical structure of the file :

File types with logical structure : structured files (text files, documents, media files), unstructured files (binary files, link libraries).
Sequential files : Files placed in the storage medium in sequence, the storage efficiency is the highest among logical files , but it is not suitable for storing variable-length files.
Index file : Invented to solve variable-length file storage, it needs to cooperate with index table storage.

Storage space allocation for auxiliary storage :

Auxiliary storage allocation methods : continuous allocation (easy and fast to read files), link allocation (implicit link and explicit link), index allocation
Storage space management of auxiliary storage : free list, free linked list, bit map.

Directory tree : Make any file or directory have a unique path.

Basic operation of Linux files : reference link

Linux file system : FAT, NTFS (improved FAT), EXT2/3/4 (extended file system, Linux file system)

6. Equipment management

The basic concept of I/O devices : external devices that input and output data to the computer;

Generalized IO devices :

Classified according to usage characteristics : storage devices (memory, disk, U disk) and interactive IO devices (keyboard, monitor, mouse);
Classified by information exchange : block devices (disks, SD cards) and character devices (printers, shell terminals);
Classified according to device sharing attributes : exclusive device, shared device, virtual device;
According to the transmission rate classification : low-speed equipment, high-speed equipment;

Buffer for IO devices : reduce the frequency of CPU processing IO requests and improve the parallelism between CPU and IO devices .

SPOOLing technology : virtual device technology, which changes synchronous calls to low-speed devices into asynchronous calls , adds a queuing dump link (input well, output well) between input and output, and SPoOLing is responsible for the connection between input (output) wells and low-speed devices Scheduling, logically, the process directly interacts with the high-speed device, reducing the waiting time of the process.

7. Implement a thread pool that supports asynchronous tasks

Thread pool : The thread pool is a container for storing multiple threads. After the CPU schedules the threads to execute, the threads will not be destroyed, and the threads will be put back into the thread pool for reuse.

Reasons to use thread pool :

Threads are scarce resources and should not be created and destroyed frequently;
Architecture decoupling, business creation and business processing decoupling, more elegant;
A thread pool is the best practice for using threads.

Implement thread-safe queue Queue

Queue : used to store multiple elements, it is a "pool" for storing various elements.
The basic functions realized : get the number of elements in the current queue, put elements into the queue, and take out elements from the queue.
Note : The queue may have multiple threads operating at the same time, so thread safety needs to be guaranteed, as follows:

Realize the basic functions of the basic task object Task
: task parameters, task unique tag (UUID), task-specific execution logic

Implement the task processing thread ProcessThread : The task processing thread needs to continuously fetch tasks from the task queue for execution, and the task processing thread needs to have a mark to indicate when the thread should stop.
The basic functions realized: basic attributes (task queue, mark), logic of thread execution (run), thread stop (stop).

Implement the task processing thread pool Pool : Store multiple task processing threads, be responsible for the start and stop of multiple threads, manage the submission of tasks to the thread pool, and send them to the threads for execution.
The basic process of implementation: basic attributes, submitting tasks (put, batch_put), thread start and stop (start, join), thread pool size (size).

Realize asynchronous task processing AsyncTask : Add a mark to the task. After the task is completed, it will be marked as completed; when the task is completed, the task running result can be obtained directly; when the task is not completed, the obtaining task result will block the obtaining thread.
Two main functions: set the running result (set_result) and get the running result (get_result)

Summary of computer operating system knowledge points (this article is enough!!!)