Things to know
This API requires Android API level 19 or higher. Make sure your app's build file uses a minSdkVersion value of at least 19.
-
Be sure to add Google's Maven code base in the and sections of your project-level
build.gradlefile .buildscriptallprojects -
Add the dependency of the Machine Learning Suite Library for Android to the module's app-level Gradle file (usually
app/build.gradle). Choose one of the following dependencies based on your needs:To bundle a model with your app::
dependencies { // ... // Use this dependency to bundle the model with your app implementation 'com.google.mlkit:face-detection:16.1.5' }To use the model in Google Play services::
dependencies { // ... // Use this dependency to use the dynamically downloaded model in Google Play Services implementation 'com.google.android.gms:play-services-mlkit-face-detection:17.1.0' } -
If you choose to use models in Google Play services, you can configure your app to automatically download models to the device after your app is installed from the Play Store. To do this, add the following declaration to your app's
AndroidManifest.xmlfile:<application ...> ... <meta-data android:name="com.google.mlkit.vision.DEPENDENCIES" android:value="face" > <!-- To use multiple models: android:value="face,model2,model3" --> </application>You can also explicitly check model availability and request downloads through the Google Play Services ModuleInstallClient API .
If you do not enable install-time model downloading or request an explicit download, the model is downloaded the first time you run the detector. Your request before the download is complete will produce no results.
Enter picture guide
For face recognition, the image size you use should be at least 480x360 pixels. In order for a machine learning kit to accurately detect faces, the input image must contain a face represented by sufficient pixel data. Typically, each face to be detected in an image should be at least 100x100 pixels. If you want to detect face outlines, the machine learning kit requires a higher resolution input: each face should be at least 200x200 pixels.
If you are detecting faces in a real-time application, you may also want to consider the overall size of the input image. Smaller images are processed relatively quickly, so to reduce latency capture the image at a lower resolution, but keep in mind the above accuracy requirements and make sure the subject's face occupies as large a part of the image as possible. See alsoTips for improving real-time performance.
Poorly focused images can also affect accuracy. If you don't get satisfactory results, ask the user to recapture the image.
The orientation of a person's face relative to the camera also affects the facial features detected by the machine learning kit. SeeFace Detection Concepts.
1. Configure the face detector
If you want to change the face detector's default settings before applying face detection to an image, specify these settings using the FaceDetectorOptions object . You can change the following settings:
| set up | |
|---|---|
| setPerformanceMode | PERFORMANCE_MODE_FAST（默认） | PERFORMANCE_MODE_ACCURATE When detecting faces, should you pay more attention to speed or accuracy? |
| setLandmarkMode | LANDMARK_MODE_NONE（默认） | LANDMARK_MODE_ALL Whether to try to identify facial "feature points": eyes, ears, nose, cheeks, mouth, etc. |
| setContourMode | CONTOUR_MODE_NONE（默认） | CONTOUR_MODE_ALL Whether to detect outlines of facial features. Only the outlines of the most prominent faces in the image are detected. |
| setClassificationMode | CLASSIFICATION_MODE_NONE（默认） | CLASSIFICATION_MODE_ALL Whether to group faces into different categories (e.g. "smiling" and "eyes open"). |
| setMinFaceSize | float（默认值：0.1f） Sets the minimum required face size, expressed as the ratio of head width to image width. |
| enableTracking | false(Default) | true Whether to assign IDs to faces for use in tracking faces across images. Note that when contour detection is enabled, only one face is detected, so face tracking will not produce useful results. For this reason, to speed up detection, do not enable contour detection and face tracking at the same time. |
For example:
// High-accuracy landmark detection and face classification FaceDetectorOptions highAccuracyOpts = new FaceDetectorOptions.Builder() .setPerformanceMode(FaceDetectorOptions.PERFORMANCE_MODE_ACCURATE) .setLandmarkMode(FaceDetectorOptions.LANDMARK_MODE_ALL) .setClassificationMode(FaceDetectorOptions.CLASSIFICATION_MODE_ALL) .build(); // Real-time contour detection FaceDetectorOptions realTimeOpts = new FaceDetectorOptions.Builder() .setContourMode(FaceDetectorOptions.CONTOUR_MODE_ALL) .build();
2. Prepare to input pictures
To detect faces in images, create a InputImage object based on the following resources on your device: Bitmap, media.Image, ByteBuffer, byte array or file. Then, pass the InputImage object to the method of FaceDetector . process
For face detection, the image size you use should be at least 480x360 pixels. Capturing frames at this lowest resolution can help reduce latency if you are detecting faces in real time.
You can create InputImage objects based on different sources. The specific methods are described below.
use media.Image
To create an InputImage object based on an media.Image object (such as when capturing an image from a device's camera), Please pass the rotation angle of the object and image to . media.ImageInputImage.fromMediaImage()
If you use the CameraX library, the OnImageCapturedListener and ImageAnalysis.Analyzer classes will Calculate the rotation angle value for you.
private class YourAnalyzer implements ImageAnalysis.Analyzer {
@Override
public void analyze(ImageProxy imageProxy) {
Image mediaImage = imageProxy.getImage();
if (mediaImage != null) {
InputImage image =
InputImage.fromMediaImage(mediaImage, imageProxy.getImageInfo().getRotationDegrees());
// Pass image to an ML Kit Vision API
// ...
}
}
}
If you are not using a camera library that provides the image rotation angle, you can calculate the rotation angle based on the device's rotation angle and the orientation of the camera sensor in the device:
private static final SparseIntArray ORIENTATIONS = new SparseIntArray();
static {
ORIENTATIONS.append(Surface.ROTATION_0, 0);
ORIENTATIONS.append(Surface.ROTATION_90, 90);
ORIENTATIONS.append(Surface.ROTATION_180, 180);
ORIENTATIONS.append(Surface.ROTATION_270, 270);
}
/**
* Get the angle by which an image must be rotated given the device's current
* orientation.
*/
@RequiresApi(api = Build.VERSION_CODES.LOLLIPOP)
private int getRotationCompensation(String cameraId, Activity activity, boolean isFrontFacing)
throws CameraAccessException {
// Get the device's current rotation relative to its "native" orientation.
// Then, from the ORIENTATIONS table, look up the angle the image must be
// rotated to compensate for the device's rotation.
int deviceRotation = activity.getWindowManager().getDefaultDisplay().getRotation();
int rotationCompensation = ORIENTATIONS.get(deviceRotation);
// Get the device's sensor orientation.
CameraManager cameraManager = (CameraManager) activity.getSystemService(CAMERA_SERVICE);
int sensorOrientation = cameraManager
.getCameraCharacteristics(cameraId)
.get(CameraCharacteristics.SENSOR_ORIENTATION);
if (isFrontFacing) {
rotationCompensation = (sensorOrientation + rotationCompensation) % 360;
} else { // back-facing
rotationCompensation = (sensorOrientation - rotationCompensation + 360) % 360;
}
return rotationCompensation;
}
Then, pass the media.Image object and rotation angle value to InputImage.fromMediaImage():
InputImage image = InputImage.fromMediaImage(mediaImage, rotation);
Use file URI
To create an InputImage object based on a file URI, pass the application context and file URI to InputImage.fromFilePath() . This is useful if you use the ACTION_GET_CONTENT intent to prompt the user to select an image from a gallery app.
InputImage image;
try {
image = InputImage.fromFilePath(context, uri);
} catch (IOException e) {
e.printStackTrace();
}
use ByteBuffer or ByteArray
If you need to create an InputImage object based on ByteBuffer or ByteArray , please First, calculate the image rotation angle according to the instructions entered previously . Then, create a object using the buffer or array and the image's height, width, color encoding format, and rotation angle:media.ImageInputImage
InputImage image = InputImage.fromByteBuffer(byteBuffer, /* image width */ 480, /* image height */ 360, rotationDegrees, InputImage.IMAGE_FORMAT_NV21 // or IMAGE_FORMAT_YV12 );
// Or:
InputImage image = InputImage.fromByteArray(
byteArray,
/* image width */480,
/* image height */360,
rotation,
InputImage.IMAGE_FORMAT_NV21 // or IMAGE_FORMAT_YV12
);
use Bitmap
If you need to create an InputImage object based on the Bitmap object, please make the following declaration:
InputImage image = InputImage.fromBitmap(bitmap, rotationDegree);
Images are represented by Bitmap objects and rotation angles.
3. Get FaceDetector instance
FaceDetector detector = FaceDetection.getClient(options); // Or use the default options: // FaceDetector detector = FaceDetection.getClient();
4. Process pictures
Pass image to process method:
Task<List<Face>> result =
detector.process(image)
.addOnSuccessListener(
new OnSuccessListener<List<Face>>() {
@Override
public void onSuccess(List<Face> faces) {
// Task completed successfully
// ...
}
})
.addOnFailureListener(
new OnFailureListener() {
@Override
public void onFailure(@NonNull Exception e) {
// Task failed with an exception
// ...
}
});
Note: If you are using the CameraX API, be sure to remove it after use ImageProxy Close (for example, add OnCompleteListener to returned by the process method). To see an example, see the VisionProcessorBase class in the Quickstart sample app. Task
5. Obtain relevant information about the detected face
If the face detection operation is successful, the system will pass a set of Face objects to the success listener. Each Face object represents a face detected in the image. For each face, you get its bounding coordinates in the input image, as well as any other information you've configured the face detector to look for. For example:
for (Face face : faces) {
Rect bounds = face.getBoundingBox();
float rotY = face.getHeadEulerAngleY(); // Head is rotated to the right rotY degrees
float rotZ = face.getHeadEulerAngleZ(); // Head is tilted sideways rotZ degrees
// If landmark detection was enabled (mouth, ears, eyes, cheeks, and
// nose available):
FaceLandmark leftEar = face.getLandmark(FaceLandmark.LEFT_EAR);
if (leftEar != null) {
PointF leftEarPos = leftEar.getPosition();
}
// If contour detection was enabled:
List<PointF> leftEyeContour =
face.getContour(FaceContour.LEFT_EYE).getPoints();
List<PointF> upperLipBottomContour =
face.getContour(FaceContour.UPPER_LIP_BOTTOM).getPoints();
// If classification was enabled:
if (face.getSmilingProbability() != null) {
float smileProb = face.getSmilingProbability();
}
if (face.getRightEyeOpenProbability() != null) {
float rightEyeOpenProb = face.getRightEyeOpenProbability();
}
// If face tracking was enabled:
if (face.getTrackingId() != null) {
int id = face.getTrackingId();
}
}
Example of face outline
When face contour detection is enabled, you get a series of points for each facial feature detected. These points represent the shape of the feature. For more information on how contours are represented, seeFace Detection Concepts.
The image below shows how these points correspond to a human face. Click on the image to enlarge it:
Real-time face detection
If you want to use face detection in a real-time application, follow these guidelines to achieve the best frame rate:
-
Configure the face detector to use face contour detection or classification and landmark detection, but not both at the same time:
Contour detection
Landmark detection
Classification
Landmark detection and classification
Contour detection and landmark detection
Contour detection and classification
Contour detection, landmark detection and classification -
Enable
FASTmode (enabled by default). -
It is recommended to capture images at a lower resolution, however, please be aware of the image size requirements for this API.
- If you use the Camera or camera2 API, then Limit the number of detector calls. If a new video frame becomes available while the detector is running, discard the frame. To see an example, see the VisionProcessorBase class in the Quickstart sample app.
- If you are using the CameraX API, make sure the Backpressure policy is set to the default ImageAnalysis .STRATEGY_KEEP_ONLY_LATEST. This ensures that only one image is served for analysis at a time. If the analyzer generates more images when it is busy, the images are automatically discarded rather than queued for delivery. After closing the image for analysis by calling ImageProxy.close(), the next latest image is delivered.
- If you want to overlay the output of the detector as a graph on the input image, first obtain the results from the machine learning suite, then complete the rendering and overlay of the image in one step. Each input frame only needs to be rendered once on the display surface. To see an example, see CameraSourcePreview and GraphicOverlay
- If you use the Camera2 API, capture the image in
ImageFormat.YUV_420_888format. If you are using an older version of the Camera API, capture the image inImageFormat.NV21format.