Table of contents
Preface
This project adopts the VGG-16 network model and uses the Kaggle open source data set to extract user features in pictures and ultimately implement the identification function of bad driving behavior on the mobile terminal.
First, by using the VGG-16 network model, this project is able to deeply learn and understand the features in driving scene images. VGG-16 is a deep convolutional neural network that is particularly suitable for image recognition tasks. Through multi-level convolution and pooling layers, it can effectively extract abstract features in images.
Secondly, the project uses open source data sets provided by Kaggle, including various driving scene images, covering different driving behaviors and conditions. Such datasets are a critical resource needed to train models.
Next, using the trained VGG-16 model, the project extracts user features in the image. Characteristics including posture, eyes, hand movements and other aspects of driving behavior can help determine whether there is bad driving behavior.
Finally, by implementing this model on the mobile side, the identification function of bad driving behavior can be deployed directly into the vehicle or driver assistance system. This real-time, mobile identification solution is expected to play a positive role in driving safety and supervision.
In general, the project combines deep learning, image processing and mobile technology to achieve intelligent identification of bad driving behaviors and provides an innovative solution to improve driving safety.
overall design
This part includes the overall system structure diagram and system flow chart.
Overall system structure diagram
The overall structure of the system is shown in the figure.
System flow chart
The system flow is shown in the figure.
The VGG-16 network architecture is shown in the figure.
Operating environment
This part includes Python environment, TensorFlow environment, Pycharm environment and Android environment.
详见行客.
Module implementation
This project includes 4 modules: data preprocessing, model construction, model training and saving, and model generation. The function introduction and related codes of each module are given below.
1. Data preprocessing
This section includes dataset sources, content, and preprocessing.
详见行客.
2. Model construction
After the data is loaded into the model, the model structure needs to be defined and the loss function optimized.
详见行客.
3. Model training and saving
After the model architecture is defined and compiled, it is trained on the training set so that the model can recognize the features of the images in the data set.
1) Model training
The code related to model training is as follows:
train_generator = train_datagen.flow_from_directory(train_data_dir, target_size=(img_height, img_width), batch_size=32, class_mode='categorical')
#读取训练集
validation_generator = train_datagen.flow_from_directory(validation_data_dir, target_size=(img_height, img_width), batch_size=32, class_mode='categorical')
#读取验证集
model.fit_generator(train_generator, samples_per_epoch=nb_train_samples, epochs=nb_epoch, validation_data=validation_generator, nb_val_samples=nb_validation_samples)
#训练模型
model.save('model+weights.h5')
#保存模型及权重
2) Model saving
The above model and weight files generated by the Keras library are in the format of .h5. In order to be read by Android programs, the .h5 files need to be converted to < /span>.pb format file, after the model is saved, it can be reused or transplanted to other environments for use.
def h5_to_pb(h5_model, output_dir, model_name, out_prefix="output_", log_tensorboard=True):
#.h5模型文件转换成.pb模型文件
if os.path.exists(output_dir) == False:
os.mkdir(output_dir)
out_nodes = []
for i in range(len(h5_model.outputs)):
out_nodes.append(out_prefix + str(i + 1))
tf.identity(h5_model.output[i], out_prefix + str(i + 1))
sess = backend.get_session()
from tensorflow.python.framework import graph_util, graph_io
#写入.pb模型文件
init_graph = sess.graph.as_graph_def()
main_graph = graph_util.convert_variables_to_constants(sess, init_graph, out_nodes)
graph_io.write_graph(main_graph, output_dir, name=model_name, as_text=False)
#输出日志文件
if log_tensorboard:
from tensorflow.python.tools import import_pb_to_tensorboard
import_pb_to_tensorboard.import_to_tensorboard(os.path.join(output_dir, model_name), output_dir)
4. Model generation
Convert the image into data, input it into the TensorFlow model and obtain the output.
1) Model import and call
This section includes the operation methods of model import and call.
a. Before writing the code to make the actual predictions, you need to add the converted model to your application's resources folder. In Android Studio, right-click "Project", jump to the Add Folder (Add Folder) section, and select AssetsFolder (Resource Folder). Create a resources folder in the application directory and copy the model into it as shown.
b. Add a new Java class to the main package of the project and name it ImageUtils. ImageUtils is an image tool class that can be used to convert and scale between Bitmap, byte, array, and Drawable image types.
The relevant code is as follows:
package com.example.doremi.testkeras2tensorflow;
import android.content.res.AssetManager;
import android.graphics.Bitmap;
import android.graphics.Canvas;
import android.graphics.Matrix;
import android.os.Environment;
import java.io.File;
import java.io.FileOutputStream;
import java.io.InputStream;
import org.json.*;
//用于处理图像的实用程序类
public class ImageUtils {
/*
*返回转换矩阵,处理裁切（如果需要保持宽高比）和旋转
*参数srcWidth为源帧的宽度
*参数srcHeight为源帧的高度
*参数dstWidth为目标帧的宽度
*参数dstHeight为目标帧的高度
*参数applyRotation为旋转的角度，为90°的整数倍
*参数maintainAspectRatio为是否维持缩放比例
*返回满足所需要求的转换
*/
public static Matrix getTransformationMatrix(
final int srcWidth,
final int srcHeight,
final int dstWidth,
final int dstHeight,
final int applyRotation,
final boolean maintainAspectRatio) {
final Matrix matrix = new Matrix();
if (applyRotation != 0) {
//进行平移，使图像中心在原点
matrix.postTranslate(-srcWidth / 2.0f, -srcHeight / 2.0f);
//绕原点旋转
matrix.postRotate(applyRotation);
}
//考虑已经应用的旋转（如果有），然后确定每个轴需要多少缩放。
final boolean transpose = (Math.abs(applyRotation) +90) % 180 == 0;
final int inWidth = transpose ? srcHeight : srcWidth;
final int inHeight = transpose ? srcWidth : srcHeight;
//必要时应用缩放
if (inWidth != dstWidth || inHeight != dstHeight) {
final float scaleFactorX = dstWidth / (float) inWidth;
final float scaleFactorY = dstHeight / (float) inHeight;
if (maintainAspectRatio) {
//按最小比例缩放，以便在保持宽高比的同时完全填充，某些图像可能会截掉边缘
final float scaleFactor = Math.max(scaleFactorX, scaleFactorY);
matrix.postScale(scaleFactor, scaleFactor);
} else {
//精确缩放
matrix.postScale(scaleFactorX, scaleFactorY);
}
}
if (applyRotation != 0) {
//从以原点为中心的参考转换回目标帧
matrix.postTranslate(dstWidth / 2.0f, dstHeight / 2.0f);
}
return matrix;
}
public static Bitmap processBitmap(Bitmap source,int size){
int image_height = source.getHeight();
int image_width = source.getWidth();
Bitmap croppedBitmap = Bitmap.createBitmap(size, size, Bitmap.Config.ARGB_8888);
Matrix frameToCropTransformations = getTransformationMatrix(image_width,image_height,size,size,0,false);
Matrix cropToFrameTransformations = new Matrix();
frameToCropTransformations.invert(cropToFrameTransformations);
final Canvas canvas = new Canvas(croppedBitmap);
canvas.drawBitmap(source, frameToCropTransformations, null);
return croppedBitmap;
}
public static float[] normalizeBitmap(Bitmap source,int size,float mean,float std){
float[] output = new float[size * size * 3];
int[] intValues = new int[source.getHeight() * source.getWidth()];
source.getPixels(intValues, 0, source.getWidth(), 0, 0, source.getWidth(), source.getHeight());
for (int i = 0; i < intValues.length; ++i) {
final int val = intValues[i];
output[i * 3] = (((val >> 16) & 0xFF) - mean)/std;
output[i * 3 + 1] = (((val >> 8) & 0xFF) - mean)/std;
output[i * 3 + 2] = ((val & 0xFF) - mean)/std;
}
return output;
}
public static Object[] argmax(float[] array){
int best = -1;
float best_confidence = 0.0f;
for(int i = 0;i < array.length;i++){
float value = array[i];
if (value > best_confidence){
best_confidence = value;
best = i;
}
}
return new Object[]{
best,best_confidence};
}
public static String getLabel( InputStream jsonStream,int index){
String label = "";
try {
byte[] jsonData = new byte[jsonStream.available()];
jsonStream.read(jsonData);
jsonStream.close();
String jsonString = new String(jsonData,"utf-8");
JSONObject object = new JSONObject(jsonString);
label = object.getString(String.valueOf(index));
}
catch (Exception e){
}
return label;
}
}
c. Add code in the main activity (main activity) that will be used to display the image and prediction results.
public void predict(final Bitmap bitmap){
//在后台线程中运行预测
new AsyncTask<Integer,Integer,Integer>(){
@Override
protected Integer doInBackground(Integer ...params){
//将图像大小调整为150*150
Bitmap resized_image = ImageUtils.processBitmap(bitmap,150);
//归一化像素
floatValues=ImageUtils.normalizeBitmap(resized_image,150,127.5f,1.0f);
//将输入传到tensorflow
tf.feed(INPUT_NAME,floatValues,1,150,150,3);
//计算预测
tf.run(new String[]{
OUTPUT_NAME});
//将输出复制到预测数组中
tf.fetch(OUTPUT_NAME,PREDICTIONS);
//获得最高预测
Object[] results = argmax(PREDICTIONS);
int class_index = (Integer) results[0];
float confidence = (Float) results[1];
try{
final String conf = String.valueOf(confidence * 100).substring(0,5);
//将预测的类别索引转换为实际的标签名称
final String label = ImageUtils.getLabel(getAssets().open("labels.json"),class_index);
//展示结果
runOnUiThread(new Runnable() {
@Override
public void run() {
progressBar.dismiss();
resultView.setText(label + " : " + conf + "%");
}
});
}
catch (Exception e){
}
return 0;
}
}.execute(0);
}
2) Related code
This section includes layout files and main activity class.
(1) Layout file
The relevant code for the layout file is as follows:
/res/layout/activity_main.xml
<?xml version="1.0" encoding="utf-8"?>
<android.support.design.widget.CoordinatorLayout xmlns:android="http://schemas.android.com/apk/res/android"
xmlns:app="http://schemas.android.com/apk/res-auto"
xmlns:tools="http://schemas.android.com/tools"
android:layout_width="match_parent"
android:layout_height="match_parent"
tools:context=".MainActivity">
<android.support.design.widget.AppBarLayout
android:layout_width="match_parent"
android:layout_height="wrap_content"
android:theme="@style/AppTheme.AppBarOverlay">
<android.support.v7.widget.Toolbar
android:id="@+id/toolbar"
android:layout_width="match_parent"
android:layout_height="?attr/actionBarSize"
android:background="?attr/colorPrimary"
app:popupTheme="@style/AppTheme.PopupOverlay" />
</android.support.design.widget.AppBarLayout>
<include layout="@layout/content_main" />
<android.support.design.widget.FloatingActionButton
android:id="@+id/predict"
android:layout_width="wrap_content"
android:layout_height="wrap_content"
android:layout_gravity="bottom|end"
android:layout_margin="@dimen/fab_margin"
app:srcCompat="@android:drawable/ic_media_play" />
</android.support.design.widget.CoordinatorLayout>
/res/layout/content_main.xml
<?xml version="1.0" encoding="utf-8"?>
<android.support.constraint.ConstraintLayout xmlns:android="http://schemas.android.com/apk/res/android"
xmlns:app="http://schemas.android.com/apk/res-auto"
xmlns:tools="http://schemas.android.com/tools"
android:layout_width="match_parent"
android:layout_height="match_parent"
app:layout_behavior="@string/appbar_scrolling_view_behavior"
tools:context=".MainActivity"
tools:showIn="@layout/activity_main">
<ScrollView
android:layout_width="match_parent"
android:layout_height="match_parent">
<LinearLayout
android:layout_width="match_parent"
android:layout_height="wrap_content"
android:orientation="vertical"
>
<TextView
android:layout_width="match_parent"
android:layout_height="wrap_content"
android:textSize="30dp"
android:layout_marginBottom="30dp"
android:text="Click the Red-Colored floating button below to show and predict the image"
/>
<ImageView
android:layout_width="match_parent"
android:layout_height="wrap_content"
android:adjustViewBounds="true"
android:scaleType="fitCenter"
android:id="@+id/imageview"
android:layout_marginBottom="10dp"
/>
<TextView
android:layout_width="wrap_content"
android:layout_height="wrap_content"
android:id="@+id/results"
/>
</LinearLayout>
</ScrollView>
</android.support.constraint.ConstraintLayout>
(2) Main activity category
The relevant code of the main activity class is as follows:
package com.specpal.mobileai;
import android.graphics.Bitmap;
import android.graphics.BitmapFactory;
import android.os.AsyncTask;
import android.os.Bundle;
import android.renderscript.ScriptGroup;
import android.support.design.widget.FloatingActionButton;
import android.support.design.widget.Snackbar;
import android.support.v7.app.AppCompatActivity;
import android.support.v7.widget.Toolbar;
import android.util.JsonReader;
import android.view.View;
import android.widget.ImageView;
import android.widget.TextView;
import android.widget.Toast;
import org.json.*;
import org.tensorflow.contrib.android.TensorFlowInferenceInterface;
import java.io.FileInputStream;
import java.io.InputStream;
public class MainActivity extends AppCompatActivity {
//加载流推理库
static {
System.loadLibrary("tensorflow_inference");
}
//模型存放路径和输入/输出节点名称
private String MODEL_PATH = "file:///android_asset/model+weights_22424.pb";
private String INPUT_NAME = "zero_padding2d_1_input";
private String OUTPUT_NAME = "output_1";
private TensorFlowInferenceInterface tf;
//保存预测的数组和图像数据的浮点值
float[] PREDICTIONS = new float[10];
private float[] floatValues;
private int[] INPUT_SIZE = {
150,150,3};
ImageView imageView;
TextView resultView;
Snackbar progressBar;
@Override
protected void onCreate(Bundle savedInstanceState) {
super.onCreate(savedInstanceState);
setContentView(R.layout.activity_main);
Toolbar toolbar = (Toolbar) findViewById(R.id.toolbar);
setSupportActionBar(toolbar);
//初始化TensorFlow
tf = new TensorFlowInferenceInterface(getAssets(),MODEL_PATH);
imageView = (ImageView) findViewById(R.id.imageview);
resultView = (TextView) findViewById(R.id.results);
progressBar = Snackbar.make(imageView,"PROCESSING IMAGE",Snackbar.LENGTH_INDEFINITE);
final FloatingActionButton predict = (FloatingActionButton) findViewById(R.id.predict);
predict.setOnClickListener(new View.OnClickListener() {
@Override
public void onClick(View view) {
try{
//从ASSETS文件夹读取图片
InputStream imageStream = getAssets().open("testimage4.gif");
Bitmap bitmap = BitmapFactory.decodeStream(imageStream);
imageView.setImageBitmap(bitmap);
progressBar.show();
predict(bitmap);
}
catch (Exception e){
}
}
});
}
//计算最大预测及其置信度的函数
public Object[] argmax(float[] array){
int best = -1;
float best_confidence = 0.0f;
for(int i = 0;i < array.length;i++){
float value = array[i];
if (value > best_confidence){
best_confidence = value;
best = i;
}
}
return new Object[]{
best,best_confidence};
}
public void predict(final Bitmap bitmap){
//在后台线程中运行预测
new AsyncTask<Integer,Integer,Integer>(){
@Override
protected Integer doInBackground(Integer ...params){
//将图像大小调整为150 x*150
Bitmap resized_image = ImageUtils.processBitmap(bitmap,150);
//归一化像素
floatValues=ImageUtils.normalizeBitmap(resized_image,150,127.5f,1.0f);
//将输入传到TensorFlow
tf.feed(INPUT_NAME,floatValues,1,150,150,3);
//计算预测
tf.run(new String[]{
OUTPUT_NAME});
//将输出复制到预测数组中
tf.fetch(OUTPUT_NAME,PREDICTIONS);
//获得最高预测
Object[] results = argmax(PREDICTIONS);
int class_index = (Integer) results[0];
float confidence = (Float) results[1];
try{
final String conf = String.valueOf(confidence * 100).substring(0,5);
//将预测的类别索引转换为实际的标签名称
final String label = ImageUtils.getLabel(getAssets().open("labels.json"),class_index);
//展示结果
runOnUiThread(new Runnable() {
@Override
public void run() {
progressBar.dismiss();
resultView.setText(label + " : " + conf + "%");
}
});
}
catch (Exception e){
}
return 0;
}
}.execute(0);
}
}
Related other blogs
Project source code download
For details, please see my blog resource download page
Download other information
If you want to continue to understand the learning routes and knowledge systems related to artificial intelligence, you are welcome to read my other blog " Heavy | Complete Artificial Intelligence AI Learning - Basics Knowledge learning route, all materials can be downloaded directly from the network disk without following any routines》
This blog refers to Github’s well-known open source platform, AI technology platform and experts in related fields : Datawhale, ApacheCN, AI Youdao and Dr. Huang Haiguang have about 100G of related information. I hope it can help all my friends.