table of Contents
1 Introduction
1.1 Python uses and advantages of the expansion modules
1.2 Design extension process
2 setup.py script
3 function interface, parameter passing, simply return value
3.1 function interface
3.2 parameter passing
3.3 simple return value
4-tuples, lists, dictionaries, buffer
5 exception handling, the reference count
5.1 throws an exception
5.2 reference count
6 GIL and multithreading
- Brief introduction
This paper records the development Python extension module used in practice, so that the reader can use low cost Python extension modules to improve application performance.
The text is not extended introductory tutorial modules, but for a certain idea of expansion modules, but has been unable to apply it effectively readers.
1.1 Python uses and advantages of the expansion modules
Python extension modules of several common uses and advantages:
Performance improvements
Python extension module using C / C ++ to write, calculate its performance is C / C ++ the same level. Performance loss on cross-language communication interface to a negligible small, it can provide very good performance support. Typical such Numpy package for scientific computing, its underlying mathematical calculations called third-party libraries, performance is the same level.
The use of multi-core computing power
By controlling the extension of the GIL, computing power may be used on multi-core CPU, not limited to single-core pure Python program limits. How many multithread core can be customized to use.
Isolation and modular system components
By each C / C ++ function provides a Python interface that is not shared between the function of any state, to achieve a good set up isolation, contribute to development and testing. And because all parameters passed by Python, easy to print and interrupted debugability has been greatly improved.
Using third-party libraries
Not every library has Python support, then you need to write your own extension module for the docking system. But modern popular large library, many have official Python extension modules, making application quality has been greatly improved, such as OpenCV and typical PyCUDA.
1.2 Design extension process
Write setup.py: script extension module configuration meta-information
Introduce the necessary header files: several header files used mainly to Python.h
Design Export function table: Exporting to use Python function table
Initialization function modules: several initialization steps necessary
- setup.py script
setup.py scripts are used to build the configuration of the expansion module. There are many simple online gives a project more practical examples and are as follows:
import platform
from distutils.core import setup,Extension
from distutils import sysconfig
cfg_vars=sysconfig.get_config_vars()
if ‘OPT’ in cfg_vars:
cfg_vars[‘OPT’]=cfg_vars[‘OPT’].replace(’-Wstrict-prototypes’,’’)
mod_expy=Extension(‘libxx.expy’,
sources=[‘libxx/expy.c’,],
libraries=[‘jpeg’,],
include_dirs=[’/opt/local/include’,],
library_dirs=[’/opt/local/lib’,],
define_macros=[(‘MAJOR_VERSION’,‘1’),(‘MINOR_VERSION’,‘0’)],
)
# Compiler to distinguish between the various platforms
if platform.system()==‘Darwin’:
os.environ[‘ARCHFLAGS’]=’-arch x86_64’
extmodlist=[mod_expy,]
elif platform.system()=‘Linux’:
os.environ[‘ARCHFLAGS’]=’-arch i386 -arch x86_64’
extmodlist=[mod_expy,]
else:
raise RuntimeError(‘Unknown system()=%s’%repr(platform.system()))
setup(name=‘libxx’,
version=‘1.0’,
description=‘libxx-python’,
author=‘gashero’,
author_email=‘[email protected]’,
packages=[‘libxx’,],
ext_modules=extmodlist,
url=‘http://example.com/’,
long_description=‘xxxxxx xxxxxx’,
)
Hello World with respect to the various levels of setup.py. As the script to add practical functions:
Provides a more comprehensive compilation parameters, such as specific macro definitions are compiled, the compiler parameters
Refer to other third-party libraries
Compile distinguish different platforms, Linux and Mac differentiated
Champions League offers you a more comprehensive meta-information
Usually we have written extension module size is not large, as setup.py script enough to use. If it comes to expansion modules more complex multiple source files, just to continue to increase the Extension objects.
- Function interface, parameter passing, simply return value
3.1 function interface
It used to define the function and passed to the Python call. It is the defining instance PyMethodDef type. An example follows:
static PyMethodDef expyMethods[]={
{“add”,expy_add,METH_VARARGS,“Add 2 number”},
{NULL, NULL, 0, NULL}
};
This definition of the four fields in the third field to invoke the function decided, optionally as follows:
METH_VARARGS: form a plurality of parameters, the most common play
METH_NOARGS: no parameter, used to output class function
METH_VARARGS | METH_KEYWORDS: have both anonymous parameters and keyword arguments
The last field is the function of the document for help () function is displayed.
3.2 parameter passing
Function is defined using the parameters containing the anonymous key parameters are different, both of the following:
static PyObject* func(PyObject* self, PyObject* args);
static PyObject* func(PyObject* self, PyObject* args, PyObject **kwargs);
If there is no corresponding contents will pass NULL. For modules function, not the object method, you can not focus on self.
Use of PyArg_ParseTuple resolution process parameters () and PyArg_ParseTupleAndKeywords (). With a string describing the parameter list, then the reference address of each incoming object.
Given the parameters and return value types may not be mutually friendly and it is generally recommended not to deliver the type of:
Relevant numeric type architecture, such as int, long, but the length of the known digital transmission type
Structure, because the location of the actual storage structure may have life cycle, the same applies to C ++ objects
Address pointer, because the architecture is related, function pointers comprising
In fact, need to pass the address of the structure, when the object is generally recommended to write objects in expansion modules, although more difficult, but the compatibility will be much better.
Do not leave anything in C / C ++ program, such as global variables.
3.3 simple return value
Function's return value must be a Python object.
Simple function returns no value can be used directly at the end of a macro function is achieved:
Py_RETURN_NONE;
This simple output function that is no problem.
You can use a little more complicated Py_BuildValue () After a string is passed in each format described values, even complex data structures can be constructed.
- Tuples, lists, dictionaries, buffer
The role of tuples in Python are immutable, which for the content on Python is very friendly, and the performance is better than a list.
The value of the list is that it can not be fixed content storage length. For example, C / C ++ library will return a lot of third-party content used in an iterative manner. At the beginning of the function does not know how many objects are able to return, only next to each call to get the next object. In this case it is suitable for creating a list, each element is then added with append () method. E.g:
PyObject * retlist = PyList_New (0); // create a queue length 0
while(pos) {
PyList_Append(retlist, Py_BuildValue("{}"));
pos=pos->next;
}
Dictionary may also be used to return the contents of a single structure. It is about C / C ++ structure of each field as an element to set the dictionary and return. Readability of this approach will be good.
Examples of the use of dict storing information, key string:
PyObject *fmtinfo=PyDict_New();
PyDict_SetItemString (fmtinfo, "index", PyInt_FromLong (1));
It is not recommended to return the list of dict, after all, consume more performance. For the return plurality of structures, a more suitable manner in list or tuple.
Use Py_BuildValue (content length that the return estimated) more suitable.
For not need to understand the content of string buffer types may be used, and performance would be more than str save memory type. str difference with respect to the type of buffer type is, buffer system does not check the contents of the same two strings. And str system will ensure there will be no repeat of the string. This calculation is to be time-consuming.
After the new buffer, a pointer to get its operating examples:
Py_ssize_t buflen=1000,_buflen;
void *bufptr;
PyObject *buf=PyBuffer_New(buflen);
if (PyObject_AsWriteBuffer(buf, &bufptr, &_buflen)<0) {
return NULL;
}
- Exception handling, the reference count
5.1 throws an exception
Used to notify Python, an error occurred in a function. Several commonly used built-in exceptions:
PyExc_ZeroDivisionError: by 0
PyExc_IOError: IO error
PyExc_TypeError: the type of error, such as the wrong type of parameters
PyExc_ValueError: error range of values
PyExc_RuntimeError: runtime error
PyExc_OSError: error when interacting with various OS
You can throw an exception to your liking. The actual method throws an exception, such as:
if (ret<0) {
PyErr_SetString(PyExc_RuntimeError,“Runtime failed!”);
return NULL;
}
Sometimes I hope thrown exception contains some parameters, such as error codes, to facilitate better debugging. You can use the following method:
if (ret<0) {
PyErr_Format(PyExc_OSError, “ERROR[%d]=%s”,
errno, strerror(errno));
}
A variety of C / C ++ library makes extensive use of the wrong code, so use the above method is widely used. See function error exceptions thrown in Python and contains the error code is usual to write pure C / C ++ program to move the crash and more beautiful. The process of writing extension modules makes this work standardized.
5.2 reference count
Extended reference count is very magical, and it is a problem widely criticized module.
Actual use, each Python API will indicate their operation on the reference count in the document. Borrowing is not changing the reference count, and the reference is to increase 1.
To avoid nausea every time to find content, could use some friendly functions. For example Py_BuildValue () is passed directly to C / C ++ data, within which is responsible for generating complex data structures, and manage the reference count.
tuple, buffer reference count management is also simpler than the list, str should be recommended.
- GIL and multithreading
Python GIL is to limit the use of multi-core direct cause, the root cause is internal Python interpreter has some global variables, such as the typical exception handling. And there are a lot of third-party modules in the Python API and use these global variables that can not be improved GIL has been progress.
Level in the expansion module is the GIL released, so that the CPU control back to the Python, and the current C / C ++ code can also continue. But it is important to note that calling any Python API must be carried out under the control of GIL. So release GIL before perform computationally intensive tasks, after the completion of the calculation, re-apply GIL, then the return value and exception handling.
The first application for release and GIL written in the function expansion module in:
static PyObject *fun(PyObject *self, PyObject *args) {
//…
PyThreadState *_save;
_save=PyEval_SaveThread();
block();
PyEval_RestoreThread(_save);
//…
}
The method also needs to call during module initialization PyEval_InitThreads ().
Another method is simple and more:
Py_BEGIN_ALLOW_THREADS;
// potentially blocking operation
Py_END_ALLOW_THREADS;
Therefore, a simple method is to use multi-core computing, the split task into a plurality of small parts, each of the small parts are placed in a running thread. Thread calls the function expansion module to calculate, calculate function releases the GIL actual calculation.
This way you can effectively use Python to manage threads, without the use of pthread such trouble in C / C ++, the only need to do the simplest of computing just fine.
