environment:
windows 10
CUDA 10.1
hidden 7.6.4
tensorflow-gpu 2.1
androidstudio 3.6
Basically it's a relatively new environment.
Because after tensorflow 2.0, I particularly like to use keras to customize models, so I want to find a way to save the model for deployment. I don’t quite understand which method of pb format model is saved, or whether all methods are saved, so I will not consider using pb model deployment for the time being. tflite is much simpler and only saves the process under the call method. The conversion process is quite simple, just go straight to the topic.
First the python part:
1. Customize a simple model with multiple inputs and multiple outputs.
class test_model2(tf.keras.Model):
def __init__(self, name="test_model2"):
super(test_model2, self).__init__(name=name)
self.conv1 = tf.keras.layers.Conv2D(filters=1, kernel_size=2, kernel_initializer=tf.ones, name=self.name + "/conv1")
@tf.function
def call(self, inputs):
output1 = self.conv1(inputs[0])
output1 = tf.squeeze(output1)
output1 = tf.reshape(output1, (1,))
output2 = self.conv1(inputs[1])
output2 = tf.squeeze(output2)
output2 = tf.reshape(output2, (1,))
return output1, output2model = test_model2()
test_input1 = tf.ones((1, 2, 2, 1))
test_input2 = tf.zeros((1, 2, 2, 1))
input_list = [test_input1, test_input2]
test_output1, test_output2 = model(input_list)
print(test_output1)
print(test_output2)After running it will print
tf.Tensor([4.], shape=(1,), dtype=float32)
tf.Tensor([0.], shape=(1,), dtype=float32)This is a fairly simple model.
2. The following is the conversion model to tflite format:
If you use a custom training loop instead of using the fit() function, you need to manually set the input size of the model. In this example, we regard it as a customized training process and just set the input size once. The input value can be random, mainly if the shape matches. Because the call function is converted by default, if training and testing do not pass through a function, it is recommended that the training function does not have the same name as the call.
test_input1 = tf.ones((1, 2, 2, 1))
test_input2 = tf.zeros((1, 2, 2, 1))
input_list = [test_input1, test_input2]
model._set_inputs(input_list)Finally convert and save the model:
converter = tf.lite.TFLiteConverter.from_keras_model(model)
tflite_model = converter.convert()
open("./save/converted_model.tflite", "wb").write(tflite_model)Now you can find the saved tflite file in the save folder.
After that comes the AndroidStudio part:
1. Create a new project
2. Modify build.gradle and add the following content
android {
...
defaultConfig {
...
ndk {
abiFilters 'armeabi-v7a', 'arm64-v8a'
}
...
}
aaptOptions {
noCompress "tflite"
}
...
}
dependencies {
...
implementation 'org.tensorflow:tensorflow-lite:2.1.0'
}3.Put in the tflite file and read it
Create an assets folder in app\src\main and put converted_model.tflite in it. This file path is not unique, it is placed in assets just for the convenience of reading.
The reading code looks like this:
String MODEL_FILE = "converted_model.tflite";
Interpreter tfLite = null;
try {
tfLite = new Interpreter(loadModelFile(getAssets(), MODEL_FILE));
}catch(IOException e){
e.printStackTrace();
}The loadModelFile function is:
MappedByteBuffer loadModelFile(AssetManager assets, String modelFilename)
throws IOException {
AssetFileDescriptor fileDescriptor = assets.openFd(modelFilename);
FileInputStream inputStream = new
FileInputStream(fileDescriptor.getFileDescriptor());
FileChannel fileChannel = inputStream.getChannel();
long startOffset = fileDescriptor.getStartOffset();
long declaredLength = fileDescriptor.getDeclaredLength();
return fileChannel.map(FileChannel.MapMode.READ_ONLY, startOffset, declaredLength);
}4. Create input tensor
The input of tflite is in ByteBuffer format:
int net_input_sz = 2;
ByteBuffer inputData1;
inputData1 = ByteBuffer.allocateDirect(net_input_sz * net_input_sz * 4);//4表示一个浮点占4byte
inputData1.order(ByteOrder.nativeOrder());
inputData1.rewind();
inputData1.putFloat(1.0f);
inputData1.putFloat(1.0f);
inputData1.putFloat(1.0f);
inputData1.putFloat(1.0f);
ByteBuffer inputData2;
inputData2 = ByteBuffer.allocateDirect(net_input_sz * net_input_sz * 4);//4表示一个浮点占4byte
inputData2.order(ByteOrder.nativeOrder());
inputData2.rewind();
inputData2.putFloat(0.0f);
inputData2.putFloat(0.0f);
inputData2.putFloat(0.0f);
inputData2.putFloat(0.0f);
Object[] inputArray = {inputData1, inputData2}; The size of the space opened by ByteBuffer is the sum of network input sizes multiplied by the bytes occupied by the precision. For example, the input shape set in this example is 1x2x2x1, so it is 4, and the floating point number occupies 4 bytes, so it is 4x4 in size.
5. Build the output tensor
float[] output1, output2;
output1 = new float[1];
output2 = new float[1];
Map<Integer, Object> outputMap = new HashMap<>();
outputMap.put(0, output1);
outputMap.put(1, output2);In our example, the output shape of the network is [1,], so we can directly construct a floating point array of size 1 here. If there is two-dimensional or even three-dimensional output, such as [2,3,4], you need to construct a multi-dimensional array new float[2][3][4]. However, I don’t like the method of constructing multi-dimensional arrays, because it is inconvenient to pass it to the Native layer for processing, so I usually reshape(output_tensor,[-1]) the output.
6. Execute inference and print output
tfLite.runForMultipleInputsOutputs(inputArray, outputMap);
Log.e("1111","output1:" + output1[0]);
Log.e("1111","output2:" + output2[0]);One sentence can solve it. The following output can be obtained
2020-02-25 16:27:55.569 22585-22585/com.stars.tflite_test E/1111: output1:4.0
2020-02-25 16:27:55.569 22585-22585/com.stars.tflite_test E/1111: output2:0.0
At this point, the conversion and deployment of the entire model is completed.
Hey, isn't it quite simple.
Attached is the package name of my java part. I forgot which ones are necessary, so I’ll just add them all:
package com.stars.tflite_test;
import android.content.res.AssetFileDescriptor;
import android.content.res.AssetManager;
import android.os.Bundle;
import com.google.android.material.floatingactionbutton.FloatingActionButton;
import com.google.android.material.snackbar.Snackbar;
import androidx.appcompat.app.AppCompatActivity;
import androidx.appcompat.widget.Toolbar;
import android.util.Log;
import android.view.View;
import android.view.Menu;
import android.view.MenuItem;
import org.tensorflow.lite.Interpreter;
import java.io.FileInputStream;
import java.io.IOException;
import java.nio.ByteBuffer;
import java.nio.ByteOrder;
import java.nio.MappedByteBuffer;
import java.nio.channels.FileChannel;
import java.util.HashMap;
import java.util.Map;
Pitfalls encountered:
1. I couldn't transfer the official BN layer to tflite (I don't know why), so I magically modified a BN layer from the official BN layer to a BN layer without so many functions, and successfully converted to tflite.
2. The result obtained by running the converted tflite model on the GPU is incorrect. I don’t know how to solve it. I have raised a question on github, but I haven’t gotten an answer yet. The problem address is https://github.com/tensorflow/tensorflow/issues/38825 . If you know anything about it, you can tell me. I tried the model written in session mode to get correct results on the GPU, but using 2. The model written in the mode of x will not work.