FIG recent experiments done associated convolution, which involves sampling FIG, the process can be abstracted as: from a node containing n, m edges FIG sampled according to a certain rule communication FIG. Since experiments using FB15k-237 data sets, contains 14,541 nodes, edges 272,115, 30,000 per sample side, the sample needs a 8S, which is the depth learning experiment unacceptable, it will lead to long GPU idle. So I started trying to use C / C ++ optimizing code, although the final effect is not optimized, but also on a learning python call C code, so in this record it.
Python source code
def get_adj_and_degrees(num_nodes, triplets):
""" Get adjacency list and degrees of the graph"""
adj_list = [[] for _ in range(num_nodes)]
for i, triplet in enumerate(triplets):
adj_list[triplet[0]].append([i, triplet[2]])
adj_list[triplet[2]].append([i, triplet[0]])
degrees = np.array([len(a) for a in adj_list])
adj_list = [np.array(a) for a in adj_list]
return adj_list, degrees
Here to get_adj_and_degrees
function as an example, we use the C / C ++ optimizing the function. This function simply play the role of presentation, specific details are not important.
C / C ++ implementation code
We sampler.hpp
optimize the function, the definition of the file is as follows:
#ifndef SAMPLER_H
#define SAMPLER_H
#include <vector>
#include "utility.hpp"
using namespace std;
// global graph data
int num_node = 0;
int num_edge = 0;
vector<int> degrees; // shape=[N]
vector<vector<vector<int>>> adj_list; // shape=[N, variable_size, 2]
void build_graph(int* src, int* rel, int* dst, int num_node_m, int num_edge_m) {
num_node = num_node_m;
num_edge = num_edge_m;
// resize the vectors
degrees.resize(num_node);
adj_list.resize(num_node);
for (int i = 0; i < num_edge; i++) {
int s = src[i];
int r = rel[i];
int d = dst[i];
vector<int> p = {i, d};
vector<int> q = {i, s};
adj_list[s].push_back(p);
adj_list[d].push_back(q);
}
for (int i = 0; i < num_node; i++) {
degrees[i] = adj_list[i].size();
}
}
#endif
Here C / C ++ function of the result is stored as a global variable is a step for after use. Specific details of the function is not telling, because our focus is on how to call a python.
Generate so library
ctypes can only call C functions, so we need to derive the C ++ function as a C function. So we lib.cpp
make the following definition:
#ifndef LIB_H
#define LIB_H
#include "sampler.hpp"
extern "C" {
void build_graph_c(int* src, int* rel, int* dst, int num_node, int num_edge) {
build_graph(src, rel, dst, num_node, num_edge);
}
}
#endif
Then use the following command to compile, in order to optimize the code, added O3
, march=native
options:
g++ lib.cpp -fPIC -shared -o libsampler.so -O3 -march=native
Python calls the C / C ++ function
After compiling, created under the current directory in libsampler.so
the library, we can write python code to call C / C ++ function of, Python code is as follows:
import numpy as np
import time
from ctypes import cdll, POINTER, Array, cast
from ctypes import c_int
class CPPLib:
"""Class for operating CPP library
Attributes:
lib_path: (str) the path of a library, e.g. 'lib.so.6'
"""
def __init__(self, lib_path):
self.lib = cdll.LoadLibrary(lib_path)
IntArray = IntArrayType()
self.lib.build_graph_c.argtypes = (IntArray, IntArray, IntArray, c_int, c_int)
self.lib.build_graph_c.restype = None
def build_graph(self, src, rel, dst, num_node, num_edge):
self.lib.build_graph_c(src, rel, dst, num_node, num_edge)
class IntArrayType:
# Define a special type for the 'int *' argument
def from_param(self, param):
typename = type(param).__name__
if hasattr(self, 'from_' + typename):
return getattr(self, 'from_' + typename)(param)
elif isinstance(param, Array):
return param
else:
raise TypeError("Can't convert %s" % typename)
# Cast from array.array objects
def from_array(self, param):
if param.typecode != 'i':
raise TypeError('must be an array of doubles')
ptr, _ = param.buffer_info()
return cast(ptr, POINTER(c_int))
# Cast from lists/tuples
def from_list(self, param):
val = ((c_int) * len(param))(*param)
return val
from_tuple = from_list
# Cast from a numpy array
def from_ndarray(self, param):
return param.ctypes.data_as(POINTER(c_int))
to sum up
python ctypes library calls using the C / C ++ function itself is not difficult, but optimized code is indeed a pit, such as scientific computing library when Numpy especially optimized. Because these library itself has been heavily optimized, using their own words implemented in C ++, it is also likely to be worse than before optimization.