One, background
What is 1.1 AoE
AoE (AI on Edge) is an open source terminal side AI bit Integrated Runtime Environment (IRE). To "stability, ease of use, security" as the design principles will help developers depth learning algorithms different frame of easily deployed to the terminal efficient execution, Github address is https://github.com/didi/aoe
Why are there two to do when running an AI terminal integrated framework, because:
First, with the rapid development of artificial intelligence, the past two years there have been many terminals running reasoning framework, giving developers more choices, but also increases the cost to deploy the AI terminal;
Second, direct access by AI reasoning framework process more complicated, involving a dynamic library access, resource loading, pre-processing and post-processing, resource release, the model upgrade, and how to protect the stability problem.
Currently AoE SDK has been used in drops of bank card use OCR, think more clearly understand the relationship between AoE and reasoning framework, host App can be integrated by the following diagram of business to understand it.
1.2 List of the terminal frame reasoning
Below are eight kinds of mainstream reasoning framework terminal running (in alphabetical order).
| No. | name | Developers | Open Source Time | description |
|---|---|---|---|---|
| 1 | TensorFlow Lite | 2017 | TensorFlow Lite using Android Neural Networks API, called by default CPU, the latest version has support for GPU. | |
| 2 | Core ML | Apple | 2017 | Core ML 2017 at WWDC Apple company released simultaneously with iOS11 end mobile machine learning framework underlying the use of Accelerate and Metal are calling CPU and GPU. Core ML you will need trained model into Core ML model |
| 3 | Caffe2 | 2017 | Caffe2 facebook framework released in 2017, a cross-platform, not only supports Windows, Linux, Macos three desktop systems, but also support mobile end iOS, Android, can be said is training and reasoning in one. | |
| 4 | NCNN | Tencent | 2017 | NCNN 2017 Tencent excellent map mobile laboratory end open source framework, using C ++ implementation, support Android and iOS two platforms. |
| 5 | Paddle-Mobile | Baidu | 2017 | Paddle-Mobile is a mobile depth study of open source end of 2017 under the framework of Baidu PaddlePaddle organization, then called mobile-deep-learning (MDL). Support Android and iOS platforms, CPU and GPU use, provide quantitative tools. |
| 6 | QNNPACK | 2018 | Facebook QNNPACK is released in 2018 int8 low quantization precision high-performance open source framework, stands Quantized Neural Network PACKage, used to accelerate the end of the phone neural network computing, has been integrated into the PyTorch 1.0, it can be used directly in the Caffe2 years. | |
| 7 | MACE | Millet | 2018 | MACE is millet 2018 announced the open source movement end in the framework of the World Summit Open Source China Open Source in order to compile and OpenCL as the underlying operator, provides a heterogeneous acceleration can easily run on different hardware models, while supporting a variety of frame model transformation. |
| 8 | MNN | Alibaba | 2019 | MNN is 2019 Ali open source framework for mobile end, do not rely on third parties Computing Base, use the compilation includes core operations, support Tensorflow, Caffe, ONNX other mainstream model file formats, support for CNN, RNN, GAN and other common network. |
1.3 AoE how to support a variety of reasoning framework
In essence, no matter what the reasoning framework, necessarily contains the following five processes, these abstract reasoning, is the basis for all kinds of reasoning AoE support framework.
Currently, AoE realized Lite supports two kinds of reasoning framework NCNN and TensorFlow, with both reasoning framework, for example, to explain the reasoning process in the form of five in the framework of their reasoning.
| Reasoning framework | The beginning of | Pretreatment | Execution reasoning | After-treatment | Release resources |
|---|---|---|---|---|---|
| NCNN | int load_param(const unsigned char* mem);int load_model(const unsigned char* mem); | ... | int input(const char* blob_name, const Mat& in);int extract(const char* blob_name, Mat& feat); | ... | void release(); |
| TF Lite | public Interpreter(@NonNull ByteBuffer byteBuffer, Interpreter.Options options) | ... | public void run(Object input, Object output) | ... | public void close(); |
1.4 What platforms AoE support
Currently, AoE has been open source runtime environment SDK includes Android and iOS platforms, in addition to Linux platform runtime environment and SDK has officially meet you.
Second, the principle
2.1 abstract reasoning process framework
As already described, different reasoning framework contains the common process, they are at the beginning of the pre-processing, execution reasoning, post-processing, release resources. For AoE integrated operating environment, the basic operation is abstract reasoning, by relying inverted design, the business depends only on the upper layer of abstraction AoE, without being concerned about access specific reasoning framework for implementation. The greatest advantage of this design is that developers can always add new framework for reasoning, without modifying the framework to achieve, so that the business development and AoE SDK to develop completely decoupled.
In AoE SDK in this abstract is InterpreterComponent (used to make the process for the initial model of, performing inference and release of resources) and Convertor (for postprocessing model input and model output pre-processing), InterpreterComponent specific implementation is as follows:
/ **
* translation component model
* /
interface InterpreterComponent <TInput, TOutput> the extends the Component {
/ **
* initialize, reasoning model resource loading frame
*
* @param context context with the service binding
* @param modelOptions model configuration list
* @return reasoning loading frame
* /
Boolean the init (@NonNull the context context, @NonNull List <AoeModelOption> modelOptions);
/ **
* performs inference operations
*
* @param iNPUT input data traffic
* @return service output data
* /
@Nullable
TOutput RUN (@NonNull TInput the INPUT);
/ **
* release resource
* /
void release ();
/ **
* Model correctly loaded
*
* @return to true, the correct loading model
* /
Boolean the isReady ();
}
Convertor specific implementation is as follows:
Convertor interface <TInput, TOutput, TModelInput, TModelOutput> {
/ **
* data preprocessing, the input data into input data model
*
* @param INPUT input data traffic
* @return model input data
* /
@Nullable
TModelInput preProcess (@ INPUT TInput give to the nonnull);
/ **
* after processing, converting the data into the service model output output data
*
* @param modelOutput model output data
* @return
* /
@Nullable
TOutput postProcess (@Nullable TModelOutput modelOutput);
}
2.2 guarantee stability
As we all know, an important issue is the Android platform development model adaptation, especially those containing large number of scenes Native operation model adaptation issues are particularly important, once the application crashes in a given model above, caused the damage is huge experience of. Data show that, because of performance problems, the loss of the mobile App daily active users accounted for 5%, the loss of these users, 60% of users chose to remain silent, no longer using the application, 3 to user switched to competing products, the rest of the user directly uninstall applications. Therefore, for a large user base of mobile applications, to ensure that any time App availability is a main flow of the most basic, the most important thing. Combined with AI reasoning process point of view, inevitably, there will be a lot of action takes place in Native process, not just speculations, there are some pre-treatment and resource recovery operations are also more prone to compatibility problems. To this end, AoE Runtime Environment SDK to develop a mechanism independent process for the Android platform for Native operations run in a separate process, while ensuring the stability of the reasoning (accidental crash will not affect the subsequent reasoning operations) and the main process stability (the main process at any time does not crash).
The specific implementation process has three main components: registered independent process, abnormal rebinding process and inter-process communication optimization.
The first part, independent registration process, an increase in the Manifest RemoteService assembly code is as follows:
<application>
<service
android:name=".AoeProcessService"
android:exported="false"
android:process=":aoeProcessor" />
</application>
The second part, an exception to rebind the independence process, in the reasoning, if found RemoteService terminated, the implementation of "bindService ()" method, restart RemoteService.
@Override
public RUN Object (Object @NonNull INPUT) {
IF (isServiceRunning ()) {
... (Code shown) performs inference //
} the else {
bindService (); // process restarted independently
}
return null;
}
The third part, cross-process communication optimization, because the independence process, necessarily involves cross-process communication, inter-process communication in the biggest problem is time-consuming loss, here, there are two factors have contributed to the loss of time-consuming:
- Transmission time-consuming
- Serialization / de-serialization time-consuming
Use binder mechanism as compared to the transmission time consuming, serialization / deserialization accounted for more than 90% of the entire communication time consuming. Thus, the serialization / de-serialization of optimization is to optimize cross-process communication priorities.
Compared the current mainstream serialization / de-serialization tool, the ultimate use AoE integrated operating environment kryo library serialization / de-serialization. The following is a comparison of the results, the data reference oschina article "performance comparison test results of various Java serialization library."
Three, MNIST integration example
3.1 of inheritance of TensorFlowLiteInterpreter
当我们要接入一个新的模型时，首先要确定的是这个模型运行在哪一个推理框架上，然后继承这个推理框架的InterpreterComponent实现，完成具体的业务流程。MNIST是运行在TF Lite框架上的模型，因此，**我们实现AoE的TF Lite的Interpreter抽象类，将输入数据转成模型的输入，再从模型的输出读取业务需要的数据。**初使化、推理执行和资源回收沿用TensorFlowLiteInterpreter的默认实现。
public class MnistInterpreter extends TensorFlowLiteInterpreter<float[], Integer, float[], float[][]> {
@Nullable
@Override
public float[] preProcess(@NonNull float[] input) {
return input;
}
@Nullable
@Override
public Integer postProcess(@Nullable float[][] modelOutput) {
if (modelOutput != null && modelOutput.length == 1) {
for (int i = 0; i < modelOutput[0].length; i++) {
if (Float.compare(modelOutput[0][i], 1f) == 0) {
return i;
}
}
}
return null;
}
}
3.2 运行时环境配置
接入MNIST的第二个步骤是配置推理框架类型和模型相关参数，代码如下：
mClient = new AoeClient(requireContext(), "mnist",
new AoeClient.Options()
.setInterpreter(MnistInterpreter.class)/*
.useRemoteService(false)*/,
"mnist");
3.3 推理执行
以下是MINST初使化推理框架、推理执行和资源回收的实现：
//初使化推理框架
int resultCode = mClient.init();
//推理执行
Object result = mClient.process(mSketchModel.getPixelData());
if (result instanceof Integer) {
int num = (int) result;
Log.d(TAG, "num: " + num);
mResultTextView.setText((num == -1) ? "Not recognized." : String.valueOf(num));
}
//资源回收
if (mClient != null) {
mClient.release();
}
Fourth, join us
Help AI landing in the terminal, an open source AoE integrated operating environment is the first step out of! Future, provide for the development of terminals to support more reasoning framework, provide more valuable properties, is our unswerving pursuit of the goal. If you, if you have thoughts on AI terminal operational environment, have questions, if you are interested in using this project, we invite you to join us.