We then artificial intelligence - smart and creative platform architecture growth path (b) - Big Data architecture articles continue
Earlier we talked about many of them are creative design platform application layer, but in fact, artificial intelligence, the most important is the algorithm, a lot of framework for the algorithm, which will lead to the realization of the underlying algorithms languages will be very much our most popular the language is python, followed by the C or C ++, as well as algorithms go language, then how do these algorithms engineered language service package it? Can not provide a bunch of arithmetic functions to the application layer platform to use it, and the application layer are generally platform java language to achieve, then the application layer platform how to cross-language calling algorithm it? And general R & D team, the staff are mostly java, java developer then how to write their doctoral research algorithms of arithmetic functions to be packaged into services?
1, the service of python algorithm
In view of this situation, it should be relatively simple, because a lot of python based web framework, we can easily put algorithm code for a service package python, the most commonly used framework flask and Django, here we flask as embodiment, to achieve a look sample code.
# -*- coding: utf-8 -*-
from flask import Flask, request, Response
import json
app = Flask(__name__)
class Algorithm(object):
…
@app.route('/getSyncCrawlSjqqResult',methods = ['GET'])
def getAlgorithm Result():
…
return Response(json.dumps(Algorithm.parser(request.args.get("para"))),mimetype="application/json")
if __name__ == '__main__':
app.run(port=3001,host='0.0.0.0',threaded=True)
2, C and C ++ as a service
A, using java JNI interface mode is called C / C ++, JNI is an acronym for Java Native Interface by using the Java Native Interface written procedures to ensure that the code can be easily ported on different platforms. From Java1.1 start, JNI become part of the standard java platform, which allows Java code and other code written in language interact. JNI beginning to native compiled language, especially designed for C and C ++, but it does not prevent you from using other programming languages, as long as the calling convention supported it. Use to interact with the local java code compiled, typically lost platform portability. However, in some cases doing so it is acceptable, even necessary. For example, using some of the old library, interact with the hardware, operating system, or to improve the performance of the program. Andrews (Android) calls the C / C ++ is used in this way a lot of, the following steps:
1), write java class methods with native declared
public class AlgorithmExample { public static native String callAlgorithm (); // All modifications are native keyword statement to local static { System.loadLibrary ( "Algorithm.so"); // load a local library of algorithms } public static void main (String [] args) { System.out.println (AlgorithmExample. callAlgorithm ()) } }
In this code, the final is at class initialization is required by System.loadLibrary ( "Algorithm.so") to load the library package so the algorithm, so here's algorithm library package is a dynamic-link library.
2), after the java code written, we need to put java code compiler generated class file
javac AlgorithmExample.java
3) generating a file name extension header h may be performed javah AlgorithmExample
jni HelloWorld header file contents:
/*DO NOT EDIT THIS FILE - it is machine generated*/ #include <jni.h> /*Header for class AlgorithmExample */ #ifndef _Included_ AlgorithmExample #define _Included_ AlgorithmExample #ifdef __cplusplus extern "C" { #endif /* *Class: AlgorithmExample *Method: callAlgorithm *Signature:()V */ JNIEXPORT String JNICALL Java_ AlgorithmExample_ callAlgorithm(JNIEnv*, jobject); #ifdef __cplusplus } #endif #endif
4), the same way to write native methods to achieve and javah command generated by the first document which stated the name of the method
#include "jni.h" #include " AlgorithmExample.h" //#include otherheaders JNIEXPORT String JNICALL Java_ AlgorithmExample_ callAlgorithm(JNIEnv *env, jobject obj) { printf("Helloworld!\n"); return ‘Helloworld’; }
5), generate dynamic link library
gcc -Wall -D_JNI_IMPLEMENTATION_ -Wl, - kill-at -Id: / java / include -Id: / java / include / win32 -shared -o (dll output file name, such as AlgorithmExample.dll) (input c / c ++ source files, such as abc.c)
6), when used, the classpath directory needs to be generated dll file (windows environment) or so files (under linux file) into the project.
B, C / C ++ callback java, that is, C / C ++ code can also be invoked via JNI ways java code, but this use is not very common, specific usage can refer to the blog in the garden of this document: HTTPS : //www.cnblogs.com/jiangjh/p/10991365.html
Whether C / C ++ JNI invocation by way of java or java call the C / C ++ JNI by the way, is prone to some problems in practice, especially java call the C / C ++ through JNI approach.
l memory leaks:
C / C ++ itself opened GC collector memory, JVM virtual machine can not help their automatic recovery, if the C / C ++ is not timely free memory, it will be a memory leak, and this memory leak by jmap obtain heap dump to see memory usage can not see this snapshot of memory usage.
l JNI some of their own problems:
I once encountered direct ByteBuffer memory can not be recovered by direct ByteBuffer jni allocated outside the virtual machine memory, the default JVM startup is not done size limit, direct ByteBuffer can -XX: to set MaxDirectMemorySize, this parameter meaning that when the outer Direct ByteBuffer allocated heap memory reaches a specified size, i.e. the trigger Full GC. Note that this value is an upper limit, the default is 64M, up to sun.misc.VM.maxDirectMemory), can be obtained -XX (in the program: the value MaxDirectMemorySize set, and the piece that can not be viewed by direct ByteBuffer jmap fast memory usage. We can only see it through the top of memory usage. About this may refer to the author of another blog post: https://www.cnblogs.com/laoqing/p/10380536.html
C, by calling python C / C ++, the algorithm then be encapsulated into python service
(Author of the original article, reproduced shall indicate the source. Original article reserved by the authors, welcome to reprint, but retain the copyright to reprint the original article bloggers, not marked provenance, author copyright will be prosecuted according to the law, please respect the author's achievement Please indicate the source: https: //www.cnblogs.com/laoqing/p/11364435.html)
Python ctypes provided, using ctypes can easily call the C language code, and ctypes C language module provides compatible data types and functions to load or so dll file.
The following C code provides two functions, one is of type int two numbers together, is a float type two numbers together, and then we use the python ctypes This code module to call
#include <stdio.h> … int add_int(int, int); float add_float(float, float); int add_int(int numA, int numB) { return numA + numB; } float add_float(float numA, float numB) { return numA + numB; } …
Use gcc -shared -Wl, -soname, adder -o adderExample.so -fPIC addExample.c to generate so files under Linux, so pack files into the python project.
Then we can write a python code to call the package so
import ctypes adderExample = ctypes.cdll.LoadLibrary('./adderExample.so ') res_int = adderExample.add_int(10,5) print("10 + 5 等于 " + str(res_int)) a = ctypes.c_float(7.2) b = ctypes.c_float(5.3) add_float = adderExample.add_float add_float.restype = ctypes.c_float print("7.2 + 5.3 等于 " + str(add_float(a, b)))
When using ctypes have significant limitations for such other similar type Boolean, and floating point, you must use the correct type ctype only can, but the object C in the call, it is difficult to achieve.
As the python interpreter itself is to use C language to achieve, in fact, so long as we algorithm code written in C can be integrated into it in accordance with the specifications python interpreter.
To be continued ...