This article will provide some tools to optimize your code. Will make the code more concise, or faster.
Of course, these are not a substitute for algorithm design, but they can still speed up Python many times. For example, deque is suitable for two-way queues, and bisect and heapq are used under appropriate conditions to improve the performance of the algorithm. And as mentioned earlier, Python provides the most advanced and efficient sorting algorithm (list.sort) today.
There is also a versatile and fast hash table (dict). And if you write iterator wrappers, functional code, or some kind of additional extensions, maybe CyToolz can be used. Of course, in the itertools and functools modules, there are many functions that can bring very efficient code.
This article mainly talks about optimizing the code of a single processor. The following will introduce some efficient function implementations, packaged extension modules, and a faster Python interpreter.
Of course, the multi-processor version can greatly improve the operating efficiency. If you want to learn about multi-core programming, you can start with the multiprocessing module. And you can also find a lot of third-party tools about distributed computing. Here you can take a look at the content about Parallel Processing on the Python wiki.
Next, I will talk about the menu of Python acceleration tools.
NumPy, SciPy, Sage, and Pandas
Let me talk about NumPy first. Its core is the implementation of a multidimensional numeric array. In addition to this data structure, several functions and operators are implemented to perform array operations efficiently. And the number of calls is simplified. It can be used to perform extremely efficient mathematical operations.
Both SciPy and Sage have NumPy built in as part of themselves, along with various other tools built in for specific scientific, mathematical, and high-performance computing modules.
Pandas is a tool focused on data analysis. If you are dealing with a large amount of semi-structured data, you may also use Pandas-related tools, such as Blaze.
PyPy、Pyston、Parakeet、Psyco 和 Unladen Swallow
The least invasive way to make your code run faster is to use a just-in-time compiler (JIT compilation). In the past, we could install Psyco directly. Import psyco after installation, then call psyco.full(). The code execution speed can be significantly improved. When running Python code, it can monitor the program in real time and compile part of the code into machine code.
Now many accelerator projects such as Psyco have stopped maintaining, but similar functions have been inherited in PyPy.
In order to facilitate analysis, optimization and translation, PyPy re-implemented Python in Python language, so that it can be JIT compiled. And PyPy can directly translate the code to a higher performance language like C.
Unladen Swallow is a JIT compiler for Python. is a version of the Python interpreter known as the Low Level Virtual Machine (LLVM). However, this development has ceased.
Pyston is a Python JIT compiler that is closer to the LLVM platform. In many cases, it has been better than Python's implementation, but there are still many places that are not perfect.
GPULib, PyStream, PyCUDA, and PyOpenCL
These four are used in the image processing unit to achieve code acceleration. What I said earlier is to use code optimization to achieve acceleration. And these are all accelerated at the hardware level. If there is a powerful GPU, we can use the GPU to calculate, thereby reducing the precious resources of the CPU.
PyStream is a bit older. GPULib provides various forms of data calculation based on GPU.
If you use GPU to accelerate your own code, you can use PyCUDA and PyOpenCL.
Pyrex, Cython, Numba, and Shedskin
All four projects are dedicated to translating Python code to C, C++, and LLVM. Shedskin will compile the code to C++ language. The main target of Pyrex, Cython compilation is C language. Cython is also a fork of Pyrex.
Moreover, Cython has additional support for NumPy arrays.
If it is oriented to arrays and mathematical calculations, Numba is a better choice, and the corresponding LLVM code will be automatically generated when imported. The upgraded version is NumbaPro, which also provides support for GPU
SWIG、F2PY、Boost.Python
These tools can package other languages as Python modules. The first one can encapsulate the C/C++ language. F2PY can wrap Fortran. Boost.Python can encapsulate the C++ language.
SUIG only needs to start a command-line tool, enter a C or C++ header file into it, and the wrapper code will be automatically generated. In addition to Python, and can be a wrapper for other languages, such as Java and PHP.
ctypes, llvm-py and CorePy2
These modules can help us realize the operation of Python's underlying objects. The ctypes module can be used to construct compiled C objects in memory. And call the C function in the shared library. However, ctypes is already included in the Python standard library.
llvm-py mainly provides the Python interface to LLVM. in order to build the code, and then compile them. It's also possible to build its compiler in Python. Of course, it is also possible to come up with your own programming language.
CorePy2 can also be accelerated, but this acceleration is run at the assembly layer.
Weave, Cinpy, and PyInline
These three packages allow us to directly use C language or other high-level languages in Python code. Mix code and still keep it clean. You can use the multi-line feature of strings in Python code to make other code typeset according to your own style.
Other tools
If we want to save memory, we can't use JIT. Generally JIT is too memory intensive. There is a saying that is right, time and memory are often not compatible, and we always have to find their balance in engineering development.
As for other things, such as the Micro Python project, this is used on embedded devices or microcontrollers.
If you just want to work in a Python environment, and then want to use another language, you can take a look at the project Julia.