sequence
You still need to summarize your studies, otherwise you will forget them.
A topic that is often said: Python multithreading is fake multithreading. Why is python weak in multithreading?
The following information is compiled from the Internet.
Global Interpreter Lock (GIL)
The execution of Python code is controlled by the Python virtual machine (interpreter).
At the beginning of its design, Python considered that in the main loop, only one thread is executing at the same time, just like running multiple processes in a single-CPU system, multiple programs can be stored in the memory, but at any time, only one program is running on the CPU run in.
Likewise, while the Python interpreter can run multiple threads, only one thread runs in the interpreter.
Access to the Python virtual machine is controlled by the Global Interpreter Lock (GIL), which ensures that only one thread is running at a time. In a multithreaded environment, the Python virtual machine executes as follows.
- Set up the GIL.
- Switch to a thread to execute.
- run.
- Set the thread to sleep state.
- Unlock the GIL.
- Repeat the above steps again.
For all I/O-oriented programs (that call built-in operating system C code), the GIL is released before the I/O call to allow other threads to run while the thread is waiting for I/O. If a thread is not using a lot of I/O operations, it will keep using the processor and the GIL for its own time slice. That said, I/O-intensive Python programs can take advantage of multithreading more than computationally-intensive Python programs.
We all know that, for example, I have a 4-core CPU, so in this way, each core can only run one thread per unit time, and then the time slice is switched in rotation. But Python is different. It doesn't matter how many cores you have, multiple cores can only run one thread per unit of time, and then the time slice rotates. Looks incredible? But that's what the GIL does. Before any Python thread executes, it must first acquire the GIL lock, and then, every 100 bytes of code is executed, the interpreter automatically releases the GIL lock, giving other threads a chance to execute. This GIL global lock actually locks the execution code of all threads. Therefore, multithreading can only be executed alternately in Python. Even if 100 threads run on a 100-core CPU, only one core can be used. Usually the interpreter we use is the official implementation of CPython, to really take advantage of multi-core, unless you rewrite an interpreter without GIL.
which is:
- You shouldn't be writing CPU-intensive code in Python... the efficiency is there...
If you really need to use concurrent in CPU-intensive code, use the multiprocessing library. This library implements the multi thread-like API interface based on multi process, and partially implements variable sharing with pickle.
One more thing, if you don't know whether your code is CPU-intensive or IO-intensive, I will teach you a method: the multiprocessing module has a dummy sub module, which implements the multiprocessing API based on multithreading. Suppose you are using a multiprocessing Pool, which implements concurrency using multiple processes
from multiprocessing import PoolIf you change this code to the following, it will become multi-threaded to achieve concurrency
from multiprocessing.dummy import PoolRun both ways, whichever is faster will do.
In addition, there is concurrent.futures this thing, including ThreadPoolExecutor and ProcessPoolExecutor, may be simpler than multiprocessing
concurrent.futures
The foundation of concurrent.futures:
concurrent.futures of python concurrency https://blog.csdn.net/dutsoft/article/details/54728706
concurrent.futures实例：
python concurrent.futures https://www.cnblogs.com/kangoroo/p/7628092.html
Use the C language link library to solve the multi-core and multi-threading problem of cpu-intensive tasks
Is that so? There is no way we can take advantage of multiple cores in Python? of course can! The multi-process just now is a solution, and the other is to call the link library of the C language. For all I/O-oriented programs (that call built-in operating system C code), the GIL is released before the I/O call to allow other threads to run while the thread is waiting for I/O. We can write some computationally intensive tasks in C language, and then load the contents of the .so link library into Python, because the C code is executed, the GIL lock will be released, so that each core can run a thread the goal of!
Summarize
Python multi-threading is equivalent to single-core multi-threading. Multi-threading has two advantages: CPU parallelism, IO parallelism, and single-core multi-threading is equivalent to breaking an arm. So, in Python, you can use multithreading, but don't expect to make efficient use of multiple cores. If you must use multiple cores through multithreading, it can only be achieved through C extensions, but this will lose the simplicity and ease of use of Python. However, don't worry too much. Although Python cannot use multi-threading to achieve multi-core tasks, it can achieve multi-core tasks through multi-process. Multiple Python processes have their own GIL locks, which do not affect each other.