[Development Language] Detailed explanation of the interoperability between C language and Python

The blogger has not authorized any person or organization to reprint any of the blogger's original articles. Thank you for your support for originality!
Blogger link

I work for an internationally renowned terminal manufacturer and am responsible for modem chip research and development.
In the early days of 5G, he was responsible for the development work related to the terminal data business layer and core network. He is currently leading the research on 6G computing network technology standards.


The blog content mainly focuses on:
       5G/6G protocol explanation,
       computing power network explanation (cloud computing, edge computing, terminal computing),
       advanced C language explanation,
       Rust language explanation

Detailed explanation of interoperability between C language and Python

Official document introduction: https://docs.python.org/zh-cn/3/extending/index.html

Since Python may define some preprocessor definitions that affect standard headers on some systems, they must be included before any standard headers are included #include<Python.h>. And it is recommended to always define it before Python.h #define PY_SSIZE_T_CLEAN.

1. C language calling Python implementation method

1.1 Calling process

1. Convert C language data to Python format;
2. Use the converted data to execute function calls to the Python interface;
3. Convert the data returned by the call from Python to C format;

1.2 Introduction to key structures and functions

Use the following function to initialize the Python environment:

PyConfig_InitPythonConfig()  # 初始化一个PyConfig对象
PyConfig_Read()              # 读取当前环境的配置信息
Py_InitializeFromConfig()    # 使能客制化的Python环境

One of the important structural diagrams is PyConfigthat the meanings of several key attributes are as follows:

• module_search_paths_set # The following variables only take effect when set to 1
• module_search_paths #Add specified search paths

1.3 Execute simple Python statements

Use the following functions to execute simple Python statements:

# 执行字符串参数中的Python语句
PyRun_SimpleString()

#例如:
PyRun_SimpleString("import sys")

1.4 Execute Python statements in files

Use the following function to execute Python statements in a file:

# 执行字符串参数中的Python语句
PyRun_SimpleFile()

# 例如:
FILE *fp = fopen("path/to/main.py", "r")
PyRun_SimpleFile(fp, "path/to/main.py")

1.5 Python module loading and library function calling

Here's how to load a Python module and call functions in the module:

PyImport_ImportModule()    # 加载指定的Python模块
PyObject_GetAttrString()   # 获取模块中的函数或者成员
PyCallable_Check()         # 检测获取的模块对象是否可以调用
PyTuple_New()              # 当从C调用Python函数的时候，入参必须使用元组封装，此函数创建一个元组对象
PyObject_CallObject()      # 调用Python函数

1.6 Conversion between C language data types and Python data types

Refer to the official website API: https://docs.python.org/zh-cn/3/c-api/stable.html

The summarized naming rules are as follows:

• Convert Python data type to C language data type
  Py<Python type>_As<C language data type>
• Convert C language data type to Python data type
  Py<Python type>_From<C language data type>

1.7 Creating Python data objects and using builtins functions

• If you want to use data types in Python, you can find
  Py<Python type>_XXX on the official website
• If you want to use Python builtins function, you can find
  Py<Python basic library>_XXX

2. Python calls C language implementation method

2.1 Calling process

1. Convert C language data types to Python format;
2. Use the converted data to execute function calls to the Python interface;
3. Convert the data returned by the call from Python to C language format;

2.2 Package C functions into modules

We need to encapsulate C variables and methods into classes (you can also define module-level methods), then package them into a module and publish them. Then Python can use C functions. Two key data structures are introduced below.

• PyModuleDef
  

  • m_base : is the base class, should always be PyModuleDef_HEAD_INIT
  • m_name : the name of the module
  • m_size : Currently set to -1, some advanced usage will use this parameter
  • m_methods : list of module methods

• PyMethodDef
  

  • ml_name : The method name as seen by Python
  • ml_meth : the corresponding C function name
  • ml_flags : indicates whether the function has parameters

2.3 How to define a class

The key data type that defines a class is PyTypeObjectthat some class attributes are defined in this type:

• tp_name : The name of the class (format is modulename.classname)
• tp_basicsize : The size of the class, used to allocate space
• tp_itemsize : 0 if it is a static class, non-0 if it is a dynamic class
• tp_flags : attribute parameters of the class, should be at least Py_TPFLAGS_DEFAULT
• tp_new : instantiation function of the class
• tp_init : Initializer of the class
• tp_dealloc : destructor of class
• tp_members : member list
• tp_methods : method list (the structure is the same as module)
• tp_getset : attribute get and set functions

A structure involving member definitions PyMemberDef, the meaning of key members:

• name : the member name as seen in Python
• type : member type
• offset : the offset of the member in the structure, obtained using the offset() function

Define the structure of the get and set methods of the attribute PyGetSetDef, and the meaning of its key members:

• name : the name of the attribute as seen by Python
• get , set : get and set methods of corresponding attributes

2.4 Define key functions of module

When a module we define is called in Python, a PyMODINIT_FUNC PyInit_<moduleName>(void)function is called. A simple PyInit_(void) implementation process is:

• Classify memory space using PyType_Ready()static classes defined for us;
• Use PyModule_Create()create module;
• Then use PyModule_AddObject()to register the class we defined into the module;

For detailed process, please see the demo below

2.5 Build a setup script and compile C language into so, pyd and other formats

# file name 'setup.py'

from distutils.core import setup, Extension

module1 = Extension('moduleName', sources = ['moduleName.c'])

setup (name = 'moduleName'
		version = '1.0'
		description = 'This is a Demo'
		ext_modules = [module1 ])

Replace moduleName in the above code with your module name, and add the corresponding C file in sources. There is no need to add a header file. Compile and install with the following commands:

python setup.py build

python setup.py install

Of course, there are many libraries now that implement python calling C language, for example

• Cython
• cffi
• ctypes
• SWIG

3. Interoperability examples between C language and Python

3.1 C language calls Python

demo.py file

def print_c_point(p)
	print(p)

main.c file

#define PY_SSIZE_T_CLEAN
#include <Python.h>

PyStatus init_python(const char *program_name, const wchar_t *additional_search_path)
{
    
    
	assert(program_name);
	
	PyStatus status;
	PyConfig config;
	PyConfig_InitPythonConfig(&config);

	status = PyConfig_SetBytesString(&config, &config.program_name, program_name);
	if(PyStatus_Exception(status)){
    
    
		goto done;
	}

	status = PyConfig_Read(&config)
	if(PyStatus_Exception(status)){
    
    
		goto done;
	}

	if(additional_search_path){
    
    
		config.module_search_paths_set = 1;
		status = PyWideStringList_Append(&config.module_search_paths, additional_search_path);
		if(PyStatus_Exception(status)){
    
    
			goto done;
		}
	}

	status = Py_InitializeFromConfig(&config);

done:
	PyConfig_Clear(&config);
	return status;
}

int main(int argc, char *argv[])
{
    
    
	init_python(argv[0], NULL);

	PyRun_SimpleString("from time import time, ctime\n"
						"print('Today is', ctime(time()))\n");

	File *fp = fopen("path/to/demo.py", "r");
	PyRun_SimpleFile(fp, "path/to/demo.py");

	PyObject *pyModule, *pyFunc;
	PyObject *pyArgs, *pyValue;
	
	pyModule = PyImport_ImportModule(demo.py);
	if(!pyModule){
    
    
		PyErr_Print();
		goto end;
	}
	
	pyFunc = PyObject_GetAttrString(pyModule, print_c_point);
	if(!pyFunc){
    
    
		Py_DECREF(pyModule);
		PyErr_Print();
		goto end;
	}

	if(PyCallable_Check(pyFunc)){
    
    
		pyArgs = PyTuple_New(1);

		for(int i=0;i < 1;++i){
    
    
			pyValue = PyLong_FromLong(3);
			if(!pyValue){
    
    
				Py_DECREF(pyArgs);
				PyErr_Print();
				goto end;
			}

			PyTuple_SetItem(pyArgs, i, pyValue);
		}

		pyValue = PyObject_CallObject(pyFunc, pyArgs);
		Py_DECREF(pyArgs);
		if(pyValue){
    
    
			printf("The result is %ld.\n". PyLong_AsLong(pyValue));
			Py_DECREF(pyValue);
		} else {
    
    
			PyErr_Print();
			goto end;
		}
	}

	Py_DECREF(pyFunc);
	Py_DECREF(pyModule);

end:
	if(Py_FinalizeEx() < 0)
		exit(-1);

	return 0;
}

3.2 Python calls C language

main.py file

import custom

if '__main__' == __name__:
	use_custom("custom module", 1234)

custom.c file

typedef struct {
    
    
	PyObject_HEAD
	PyObject *user_name;
	unsigned int passwd;
} customObject;


static int 
custom_clear(customObject *self)
{
    
    
	Py_CLEAR(self->user_name);

	return 0;
}

static void
custom_dealloc(customObject *self)
{
    
    
	PyObecjt_GC_UnTrack(self);
	custom_clear(self);
	Py_TYPE(self)->tp_free((PyObject*) self);
}

static PyObject*
custom_new(PyTypeObject *type, PyObject *args, PyObject *kwds)
{
    
    
	customObject *self;
	self = (customObject *)type->tp_alloc(type, 0);
	if(self != NULL){
    
    
		self->user_name = PyUnicode_FromString("");
		if(self->user_name == NULL){
    
    
			Py_DECREF(self);
			return NULL;
		}

		self->passwd = 1234;
	}

	return (PyObject *) self;
}

static int
custom_init(customObject *self, PyObject *args, PyObject *kwds)
{
    
    
	static char *kwlist[] = {
    
    "user_name","passwd",NULL};
	PyObject *user_name = NULL, *tmp;
	if(!PyArg_ParseTupleAndKeywords(args, kwds, "|UkI", kwlist, &user_name, &self->passwd))
		return -1;

	if(user_name){
    
    
		tmp = self->user_name;
		Py_INCREF(user_name);
		self->user_name = user_name;
		Py_DECREF(tmp);
	}
	
	return 0;
}

static PyMemberDef 
custom_members[] = {
    
    
	{
    
    "passwd", T_ULONG, offset(customObject, passwd), 0, "user password"},
	{
    
    NULL}
};

static PyObject *
custom_getusername(customObject *self, void *closure)
{
    
    
	Py_INCREF(self->user_name);
	return self->user_name;
}

static PyObject *
custom_setusername(customObject *self, PyObject *value, void *closure)
{
    
    
	if(value == NULL) {
    
    
		PyErr_SetString(PyExc_TypeError, "user name is not NULL");
		return -1;
	}

	if(!PyUnicode_Check(value)) {
    
    
		PyErr_SetString(PyExc_TypeError, "user name should string");
		return -1;
	}
	
	Py_INCREF(value);
	Py_CLEAR(self->user_name);
	self->user_name = value;

	return 0;
}

static int
custom_getpassword(customObject *self, void *closure)
{
    
    
	PyObject *tmp = PyLong_FromUnsignedLong(self->passwd);
	Py_INCREF(tmp);
	
	return tmp;
}

static int
custom_setpassword(customObject *self, PyObject *value, void *closure)
{
    
    
	if(value == NULL) {
    
    
		PyErr_SetString(PyExc_TypeError, "user password is not NULL");
		return -1;
	}

	if(!PyLong_Check(value)) {
    
    
		PyErr_SetString(PyExc_TypeError, "user password should integer");
		return -1;
	}

	self->passwd = PyLong_AsUnsignedLong(value);

	return 0;
}

static PyGetSetDef
custom_getsetters[] = {
    
    
	{
    
    "user_name", (getter)custom_getusername, (setter)custom_setusername, "user name", NULL},
	{
    
    "passwd", (getter)custom_getpassword, (setter)custom_setpassword, "user password", NULL},
	{
    
    NULL}
};

static PyObject*
custom_printUserInfo(customObject *self, PyObject *Py_UNUSED(ignored))
{
    
    
	printf("user name is %s and password is %ld.\n",self->user_name,self->passwd);
}

static PyMethodDef custom_methods[] = {
    
    
	{
    
    "custom_printUserInfo", (PyCFunction) custom_printUserInfo, METH_NOARGS, "print user info"},
	{
    
    NULL}
};


static PyTypeObject customType = {
    
    
	PyVarObject_HEAD_INIT(NULL,0)
	.tp_name = "custom.custom",
	.tp_doc = PyDoc_STR("custom object"),
	.tp_basicsize = sizeof(customObject),
	.tp_itemsize = 0,
	.tp_flags = Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE | Py_TPFLAGS_HAVE_GC,
	.tp_new = custom_new,
	.tp_init = (initproc)custom_init,
	.tp_dealloc = (destructor) custom_dealloc,
	.tp_clear = (inquiry) custom_clear,
	.tp_members = custom_members,
	.tp_methods = custom_methods,
	.tp_getset = custom_getsetters,
};

static PyModuleDef custommodule = {
    
    
	PyModuleDef_HEAD_INIT,
	.m_name = "custom",
	.m_doc = "example module that creates an extension type",
	.m_size = 1
};

PyMODINIT_FUNC
PyInit_custom(void)
{
    
    
	PyObject *m;

	if(PyType_Ready(&customType) < 0)
		return NULL;

	m = PyModule_Create(&custommodule);
	if(m == NULL) return NULL;

	Py_INCREF(&customType);
	if(PyModule_AddObject(m, "custom", (PyObject*)&customType) < 0){
    
    
		Py_DECREF(&customType);
		Py_DECREF(m);
		return NULL;
	}

	return m;
}

---