1. Frontier
The era of artificial intelligence is rapidly approaching. Among them, face recognition is a relatively popular technology at present, and it is also used more and more in China, such as face punch card, face APP, identity recognition, face access control, etc. The current face recognition technology is divided into two ways: WEBAPI and SDK calling. WEBAPI needs real-time networking, and SDK calling can be used offline.
本次使用的虹软提供的人脸识别的SDK，此SDK也可根据不同应用场景设计，针对性强。包括人脸检测、人脸跟踪、人脸识别，即使在离线环境下也可正常运行。
虹软公司是一家具有硅谷背景的图像处理公司，除了人脸技术以外，还有多项图像及视频处理技术。他们的双摄像头处理算法和人脸美化算法囊括了包括OPPO VIVO，SUMAMNG一系列手机厂商。
2. The goal of the project
We need to implement a face recognition function. To put it simply, it is the rear camera of the machine, which recognizes the face information captured in real time in the camera. If the person database has been registered, it will display the recognized face information, such as the registered name; if not, it will prompt that it is not registered.
This feature has multiple application scenarios, for example, in railway stations or in punch and access control systems.
3. The process of face recognition
Face recognition includes two essential processes, face registration and real-time recognition.
Face registration refers to registering the feature information of the face into the face information database. There are many sources of face registration, such as
- National ID Bank
- Enterprise self-built face recognition library
- big internet database
Facial feature extraction is an irreversible process, and you cannot restore a person's face photo from facial feature information.
When the online library is used, it is necessary to transfer photo information, or extract image feature values,
Offline SDKs are relatively safe, but online SDKs usually provide more access and calling methods, which should be selected according to the actual situation.
4. Define and implement the relevant functions of the face library
As mentioned earlier, we want to define our own face library. The face library is stored in List in the program and saved as a txt file in the system.
Through the query engine, you can know that the face information is stored in the AFR_FSDKFace class. The main structure of this is
public static final int FEATURE_SIZE = 22020;
byte[] mFeatureData;If we want to perform face registration, we need to define another class to associate the face information with the name.
class FaceRegist {
String mName;
List<AFR_FSDKFace> mFaceList;
public FaceRegist(String name) {
mName = name;
mFaceList = new ArrayList<>();
}
}
A byte array containing the length and content of the feature information.
We define these functions in the class FaceDB. FaceDB needs to include the functions of engine definition, initialization, saving face information in the repository and reading face information from the repository
5. Initialize the engine
For the sake of program structure, we separate the code related to face recognition into a class FaceDB, and define the necessary variables
public static String appid = "bCx99etK9Ns4Saou1EbFdC18xHdY9817EKw****";
public static String ft_key = "CopwZarSihp1VBu5AyGxfuLQdRMPyoGV2C2opc****";
public static String fd_key = "CopwZarSihp1VBu5AyGxfuLXnpccQbWAjd86S8****";
public static String fr_key = "CopwZarSihp1VBu5AyGxfuLexDsi8yyELdgsj4****";
String mDBPath;
List<FaceRegist> mRegister;
AFR_FSDKEngine mFREngine;
AFR_FSDKVersion mFRVersion;
Define a constructor with parameters to initialize the engine
public FaceDB(String path) {
mDBPath = path;
mRegister = new ArrayList<>();
mFRVersion = new AFR_FSDKVersion();
mUpgrade = false;
mFREngine = new AFR_FSDKEngine();
AFR_FSDKError error = mFREngine.AFR_FSDK_InitialEngine(FaceDB.appid, FaceDB.fr_key);
if (error.getCode() != AFR_FSDKError.MOK) {
Log.e(TAG, "AFR_FSDK_InitialEngine fail! error code :" + error.getCode());
} else {
mFREngine.AFR_FSDK_GetVersion(mFRVersion);
Log.d(TAG, "AFR_FSDK_GetVersion=" + mFRVersion.toString());
}
}Define a destructor to release system resources occupied by the engine
public void destroy() {
if (mFREngine != null) {
mFREngine.AFR_FSDK_UninitialEngine();
}
}6. Realize face adding and reading functions
Usually the face database will be stored in the database, this time we use List to perform a simple simulation and save it in a text file, read from the text when needed, and write to the file when saving.
We use the addFace method to add the face information to be registered to the face database
public void addFace(String name, AFR_FSDKFace face) {
try {
//check if already registered.
boolean add = true;
for (FaceRegist frface : mRegister) {
if (frface.mName.equals(name)) {
frface.mFaceList.add(face);
add = false;
break;
}
}
if (add) { // not registered.
FaceRegist frface = new FaceRegist(name);
frface.mFaceList.add(face);
mRegister.add(frface);
}
if (!new File(mDBPath + "/face.txt").exists()) {
if (!saveInfo()) {
Log.e(TAG, "save fail!");
}
}
//save name
FileOutputStream fs = new FileOutputStream(mDBPath + "/face.txt", true);
ExtOutputStream bos = new ExtOutputStream(fs);
bos.writeString(name);
bos.close();
fs.close();
//save feature
fs = new FileOutputStream(mDBPath + "/" + name + ".data", true);
bos = new ExtOutputStream(fs);
bos.writeBytes(face.getFeatureData());
bos.close();
fs.close();
} catch (FileNotFoundException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
}
}Read faces from file using loadFaces
public boolean loadFaces(){
if (loadInfo()) {
try {
for (FaceRegist face : mRegister) {
Log.d(TAG, "load name:" + face.mName + "'s face feature data.");
FileInputStream fs = new FileInputStream(mDBPath + "/" + face.mName + ".data");
ExtInputStream bos = new ExtInputStream(fs);
AFR_FSDKFace afr = null;
do {
if (afr != null) {
if (mUpgrade) {
//upgrade data.
}
face.mFaceList.add(afr);
}
afr = new AFR_FSDKFace();
} while (bos.readBytes(afr.getFeatureData()));
bos.close();
fs.close();
}
return true;
} catch (FileNotFoundException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
}
} else {
if (!saveInfo()) {
Log.e(TAG, "save fail!");
}
}
return false;
}7. Implement business logic
7.1 Implement face registration function
The prerequisite for face recognition is that the face information must be registered in the face database first.
The first step is of course to get the photo to be registered, we can use the camera or the photo. We use AlertDialog to pop up the select box
new AlertDialog.Builder(this)
.setTitle("请选择注册方式")
.setIcon(android.R.drawable.ic_dialog_info)
.setItems(new String[]{"打开图片", "拍摄照片"}, this)
.show();Process it in the corresponding event handler
switch (which){
case 1://摄像头
Intent getImageByCamera = new Intent("android.media.action.IMAGE_CAPTURE");
ContentValues values = new ContentValues(1);
values.put(MediaStore.Images.Media.MIME_TYPE, "image/jpeg");
mPath = getContentResolver().insert(MediaStore.Images.Media.EXTERNAL_CONTENT_URI, values);
getImageByCamera.putExtra(MediaStore.EXTRA_OUTPUT, mPath);
startActivityForResult(getImageByCamera, REQUEST_CODE_IMAGE_CAMERA);
break;
case 0://图片
Intent getImageByalbum = new Intent(Intent.ACTION_GET_CONTENT);
getImageByalbum.addCategory(Intent.CATEGORY_OPENABLE);
getImageByalbum.setType("image/jpeg");
startActivityForResult(getImageByalbum, REQUEST_CODE_IMAGE_OP);
break;
default:;
}After obtaining a photo, we need to implement the face detection function later.
if (requestCode == REQUEST_CODE_IMAGE_OP && resultCode == RESULT_OK) {
mPath = data.getData();
String file = getPath(mPath);
//TODO: add image coversion
}In the above code, we get the image data bmp we need, and take out the picture.
We implement this code in the function decodeImage in the Application class
public static Bitmap decodeImage(String path) {
Bitmap res;
try {
ExifInterface exif = new ExifInterface(path);
int orientation = exif.getAttributeInt(ExifInterface.TAG_ORIENTATION, ExifInterface.ORIENTATION_NORMAL);
BitmapFactory.Options op = new BitmapFactory.Options();
op.inSampleSize = 1;
op.inJustDecodeBounds = false;
//op.inMutable = true;
res = BitmapFactory.decodeFile(path, op);
//rotate and scale.
Matrix matrix = new Matrix();
if (orientation == ExifInterface.ORIENTATION_ROTATE_90) {
matrix.postRotate(90);
} else if (orientation == ExifInterface.ORIENTATION_ROTATE_180) {
matrix.postRotate(180);
} else if (orientation == ExifInterface.ORIENTATION_ROTATE_270) {
matrix.postRotate(270);
}
Bitmap temp = Bitmap.createBitmap(res, 0, 0, res.getWidth(), res.getHeight(), matrix, true);
Log.d("com.arcsoft", "check target Image:" + temp.getWidth() + "X" + temp.getHeight());
if (!temp.equals(res)) {
res.recycle();
}
return temp;
} catch (Exception e) {
e.printStackTrace();
}
return null;
}Call AFD_FSDK_StillImageFaceDetection to return the detected face information
For face registration, you must first detect the face. For static images, the corresponding FD in the Arcsoft Face SDK is provided, and a method name is provided, called AFD_FSDK_StillImageFaceDetection.
Let's take a look at the parameter list
Note that the AFD_FSDKFace object engine is reused internally. If you need to save it, please clone a copy of the AFD_FSDKFace object or save it separately
AFD_FSDKFace is the result of face recognition, defined as follows
public class AFD_FSDKFace {
Rect mRect;
int mDegree;
}mRect defines a rectangular frame Rect
Before that, we need to note that the image format used by Arcsoft Face SDK is the NV21 format, so we need to convert the acquired image into the corresponding format. The corresponding conversion function is provided in Android_extend.jar
byte[] data = new byte[mBitmap.getWidth() * mBitmap.getHeight() * 3 / 2];
ImageConverter convert = new ImageConverter();
convert.initial(mBitmap.getWidth(), mBitmap.getHeight(), ImageConverter.CP_PAF_NV21);
if (convert.convert(mBitmap, data)) {
Log.d(TAG, "convert ok!");
}
convert.destroy();Now we can call the AFD_FSDK_StillImageFaceDetection method
8. Draw the face frame
The position information and depth information of the detected face are saved in List<AFD_FSDKFace>.
We can draw the detected face position information with a rectangular frame on the picture to represent the detected face information.
Canvas canvas = mSurfaceHolder.lockCanvas();
if (canvas != null) {
Paint mPaint = new Paint();
boolean fit_horizontal = canvas.getWidth() / (float)src.width() < canvas.getHeight() / (float)src.height() ? true : false;
float scale = 1.0f;
if (fit_horizontal) {
scale = canvas.getWidth() / (float)src.width();
dst.left = 0;
dst.top = (canvas.getHeight() - (int)(src.height() * scale)) / 2;
dst.right = dst.left + canvas.getWidth();
dst.bottom = dst.top + (int)(src.height() * scale);
} else {
scale = canvas.getHeight() / (float)src.height();
dst.left = (canvas.getWidth() - (int)(src.width() * scale)) / 2;
dst.top = 0;
dst.right = dst.left + (int)(src.width() * scale);
dst.bottom = dst.top + canvas.getHeight();
}
canvas.drawBitmap(mBitmap, src, dst, mPaint);
canvas.save();
canvas.scale((float) dst.width() / (float) src.width(), (float) dst.height() / (float) src.height());
canvas.translate(dst.left / scale, dst.top / scale);
for (AFD_FSDKFace face : result) {
mPaint.setColor(Color.RED);
mPaint.setStrokeWidth(10.0f);
mPaint.setStyle(Paint.Style.STROKE);
canvas.drawRect(face.getRect(), mPaint);
}
canvas.restore();
mSurfaceHolder.unlockCanvasAndPost(canvas);
break;
}
}9. Register the face to the face database
When a face is detected, we can enter the corresponding description information and add it to the face database.
In order to improve the accuracy of recognition, we can register face information for a person multiple times.
public void addFace(String name, AFR_FSDKFace face) {
try {
//check if already registered.
boolean add = true;
for (FaceRegist frface : mRegister) {
if (frface.mName.equals(name)) {
frface.mFaceList.add(face);
add = false;
break;
}
}
if (add) { // not registered.
FaceRegist frface = new FaceRegist(name);
frface.mFaceList.add(face);
mRegister.add(frface);
}
if (!new File(mDBPath + "/face.txt").exists()) {
if (!saveInfo()) {
Log.e(TAG, "save fail!");
}
}
//save name
FileOutputStream fs = new FileOutputStream(mDBPath + "/face.txt", true);
ExtOutputStream bos = new ExtOutputStream(fs);
bos.writeString(name);
bos.close();
fs.close();
//save feature
fs = new FileOutputStream(mDBPath + "/" + name + ".data", true);
bos = new ExtOutputStream(fs);
bos.writeBytes(face.getFeatureData());
bos.close();
fs.close();
} catch (FileNotFoundException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
}
}Finally, don't forget to destroy the face detection engine
err = engine.AFD_FSDK_UninitialFaceEngine();
Log.d("com.arcsoft", "AFD_FSDK_UninitialFaceEngine =" + err.getCode()); 10. Realize face recognition
After the above code is ready, we can start our face recognition function. We use a third-party extension library, ExtGLSurfaceView's extension library CameraGLSurfaceView, to display detected faces and corresponding description information with ImageView and TextView.
The first is to define the layout.
<?xml version="1.0" encoding="utf-8"?>
<RelativeLayout xmlns:android="http://schemas.android.com/apk/res/android"
android:layout_width="match_parent"
android:layout_height="match_parent" >
<com.guo.android_extend.widget.CameraSurfaceView
android:id="@+id/surfaceView"
android:layout_width="1dp"
android:layout_height="1dp"/>
<com.guo.android_extend.widget.CameraGLSurfaceView
android:id="@+id/glsurfaceView"
android:layout_width="match_parent"
android:layout_height="match_parent"
android:layout_centerInParent="true"/>
<ImageView
android:id="@+id/imageView"
android:layout_width="120dp"
android:layout_height="120dp"
android:layout_marginLeft="10dp"
android:layout_marginTop="10dp"/>
<TextView
android:id="@+id/textView"
android:layout_width="wrap_content"
android:layout_height="wrap_content"
android:layout_alignLeft="@+id/imageView"
android:layout_alignRight="@+id/imageView"
android:layout_below="@+id/imageView"
android:layout_marginTop="10dp"
android:text="@string/app_name"
android:textAlignment="center"/>
<TextView
android:id="@+id/textView1"
android:layout_width="wrap_content"
android:layout_height="wrap_content"
android:layout_alignLeft="@+id/imageView"
android:layout_alignRight="@+id/imageView"
android:layout_below="@+id/textView"
android:layout_marginTop="10dp"
android:text="@string/app_name"
android:textAlignment="center"/>
</RelativeLayout>Because the image format required by the engine is NV21, the image format in the camera needs to be preset to NV21
public Camera setupCamera() {
// TODO Auto-generated method stub
mCamera = Camera.open(Camera.CameraInfo.CAMERA_FACING_BACK);
try {
Camera.Parameters parameters = mCamera.getParameters();
parameters.setPreviewSize(mWidth, mHeight);
parameters.setPreviewFormat(ImageFormat.NV21);
for( Camera.Size size : parameters.getSupportedPreviewSizes()) {
Log.d(TAG, "SIZE:" + size.width + "x" + size.height);
}
for( Integer format : parameters.getSupportedPreviewFormats()) {
Log.d(TAG, "FORMAT:" + format);
}
List<int[]> fps = parameters.getSupportedPreviewFpsRange();
for(int[] count : fps) {
Log.d(TAG, "T:");
for (int data : count) {
Log.d(TAG, "V=" + data);
}
}
mCamera.setParameters(parameters);
} catch (Exception e) {
e.printStackTrace();
}
if (mCamera != null) {
mWidth = mCamera.getParameters().getPreviewSize().width;
mHeight = mCamera.getParameters().getPreviewSize().height;
}
return mCamera;
}To recognize the face from the camera, you need to use the FT library. The FT library optimizes the face detection part in the face tracking algorithm and is a library specially optimized for video processing.
11. Initialize the face detection engine (FT)
Like FD, we need to initialize the face recognition FT engine.
Log.d(TAG, "AFT_FSDK_InitialFaceEngine =" + err.getCode());
err = engine.AFT_FSDK_GetVersion(version);
Log.d(TAG, "AFT_FSDK_GetVersion:" + version.toString() + "," + err.getCode());
In the preview event processing function of the camera, the face recognition function of FT is called first, and then the feature extraction function of face information in FR is called.
AFT_FSDKError err = engine.AFT_FSDK_FaceFeatureDetect(data, width, height, AFT_FSDKEngine.CP_PAF_NV21, result);
AFR_FSDKError error = engine.AFR_FSDK_ExtractFRFeature(mImageNV21, mWidth, mHeight, AFR_FSDKEngine.CP_PAF_NV21,mAFT_FSDKFace.getRect(), mAFT_FSDKFace.getDegree(), result);The result here saves the face feature information. We can save it or down and compare it with other information in the system.
AFR_FSDKMatching score = new AFR_FSDKMatching();
float max = 0.0f;
String name = null;
for (FaceDB.FaceRegist fr : mResgist) {
for (AFR_FSDKFace face : fr.mFaceList) {
error = engine.AFR_FSDK_FacePairMatching(result, face, score);
Log.d(TAG, "Score:" + score.getScore() + ", AFR_FSDK_FacePairMatching=" + error.getCode());
if (max < score.getScore()) {
max = score.getScore();
name = fr.mName;
}
}
}When the feature information of the score is greater than 0.6, we can consider that the face is matched. Displays face matching information.
In the above loop, you can see that it traverses the real library to search. Our purpose is to demonstrate that in practice, we can jump out of the loop after finding a face with a high matching value.