You may not have thought that CameraX is also part of Jetpack.
Our lives have become more and more inseparable from the camera, from selfies to live broadcasts, scanning codes to VR and so on. The pros and cons of cameras have naturally become a field for manufacturers to chase. For app developers, how to quickly drive the camera, provide an excellent shooting experience, and optimize the power consumption of the camera is the goal they have been pursuing.
Preface
The Camera interface was deprecated in Android 5.0, so the general approach is to use its replacement Camera2 interface. But with the advent of CameraX, this choice is no longer the only one.
Let's first review the simple requirement of image preview and how it is implemented using the Camera2 interface.
Camera2
Aside from additional processing such as callbacks and exceptions, it still requires multiple steps to achieve, which is more cumbersome. ※The code is omitted for space reasons and only explains the steps※
Similarly, if the image preview uses CameraX, the implementation is very simple.
CameraX
Image preview
It can be done in a dozen lines. Like Camera2, you need to display the preview control PreviewView on the layout and ensure that you have the camera permission. The difference is mainly reflected in the configuration steps of the camera.
private void setupCamera(PreviewView previewView) {
ListenableFuture<ProcessCameraProvider> cameraProviderFuture =
ProcessCameraProvider.getInstance(this);
cameraProviderFuture.addListener(() -> {
try {
mCameraProvider = cameraProviderFuture.get();
bindPreview(mCameraProvider, previewView);
} catch (ExecutionException | InterruptedException e) {
e.printStackTrace();
}
}, ContextCompat.getMainExecutor(this));
}
private void bindPreview(@NonNull ProcessCameraProvider cameraProvider,
PreviewView previewView) {
mPreview = new Preview.Builder().build();
mCamera = cameraProvider.bindToLifecycle(this,
CameraSelector.DEFAULT_BACK_CAMERA, mPreview);
mPreview.setSurfaceProvider(previewView.getSurfaceProvider());
}
Lens switching
If you want to switch lenses, just bind the CameraSelector example of the target lens to the CameraProvider. We add buttons on the screen to switch shots.
public void onChangeGo(View view) {
if (mCameraProvider != null) {
isBack = !isBack;
bindPreview(mCameraProvider, binding.previewView);
}
}
private void bindPreview(@NonNull ProcessCameraProvider cameraProvider,
PreviewView previewView) {
...
CameraSelector cameraSelector = isBack ? CameraSelector.DEFAULT_BACK_CAMERA
: CameraSelector.DEFAULT_FRONT_CAMERA;
// 绑定前确保解除了所有绑定，防止CameraProvider重复绑定到Lifecycle发生异常
cameraProvider.unbindAll();
mCamera = cameraProvider.bindToLifecycle(this, cameraSelector, mPreview);
...
}
Lens focus
Shooting that cannot be focused is incomplete. We listen to the Touch event of Preview and tell CameraX the touch coordinates to start focusing.
protected void onCreate(@Nullable Bundle savedInstanceState) {
...
binding.previewView.setOnTouchListener((v, event) -> {
FocusMeteringAction action = new FocusMeteringAction.Builder(
binding.previewView.getMeteringPointFactory()
.createPoint(event.getX(), event.getY())).build();
try {
showTapView((int) event.getX(), (int) event.getY());
mCamera.getCameraControl().startFocusAndMetering(action);
}...
});
}
private void showTapView(int x, int y) {
PopupWindow popupWindow = new PopupWindow(ViewGroup.LayoutParams.WRAP_CONTENT,
ViewGroup.LayoutParams.WRAP_CONTENT);
ImageView imageView = new ImageView(this);
imageView.setImageResource(R.drawable.ic_focus_view);
popupWindow.setContentView(imageView);
popupWindow.showAsDropDown(binding.previewView, x, y);
binding.previewView.postDelayed(popupWindow::dismiss, 600);
binding.previewView.playSoundEffect(SoundEffectConstants.CLICK);
}
In addition to image preview, there are many other usage scenarios, such as image capture, image analysis and video recording. CameraX abstracts these usage scenarios as UseCase.
It has four subcategories, namely Preview, ImageCapture, ImageAnalysis and VideoCapture. Next, I will introduce how to use them.
Image capture
The image can be taken with the help of takePicture() provided by ImageCapture. Support saving to external storage space, of course, you need to obtain read and write permissions for external storage.
private void takenPictureInternal(boolean isExternal) {
final ContentValues contentValues = new ContentValues();
contentValues.put(MediaStore.MediaColumns.DISPLAY_NAME, CAPTURED_FILE_NAME
+ "_" + picCount++);
contentValues.put(MediaStore.MediaColumns.MIME_TYPE, "image/jpeg");
ImageCapture.OutputFileOptions outputFileOptions =
new ImageCapture.OutputFileOptions.Builder(
getContentResolver(),
MediaStore.Images.Media.EXTERNAL_CONTENT_URI, contentValues)
.build();
if (mImageCapture != null) {
mImageCapture.takePicture(outputFileOptions, CameraXExecutors.mainThreadExecutor(),
new ImageCapture.OnImageSavedCallback() {
@Override
public void onImageSaved(@NonNull ImageCapture.OutputFileResults outputFileResults) {
Toast.makeText(DemoActivityLite.this, "Picture got"
+ (outputFileResults.getSavedUri() != null
? " @ " + outputFileResults.getSavedUri().getPath()
: "") + ".", Toast.LENGTH_SHORT)
.show();
}
...
});
}
}
private void bindPreview(@NonNull ProcessCameraProvider cameraProvider,
PreviewView previewView) {
...
mImageCapture = new ImageCapture.Builder()
.setTargetRotation(previewView.getDisplay().getRotation())
.build();
...
// 需要将ImageCapture场景一并绑定
mCamera = cameraProvider.bindToLifecycle(this, cameraSelector, mPreview, mImageCapture);
...
}
Image analysis
Image analysis refers to the real-time analysis of the previewed image to identify information such as color and content, and apply it to business scenarios such as machine learning and QR code recognition.
Continue to make some modifications to the demo and add a button to scan the QR code. Click the button to enter the scan code mode, and the analysis result will pop up after the QR code is successfully parsed.
public void onAnalyzeGo(View view) {
if (!isAnalyzing) {
mImageAnalysis.setAnalyzer(CameraXExecutors.mainThreadExecutor(), image -> {
analyzeQRCode(image);
});
}
...
}
// 从ImageProxy取出图像数据，交由二维码框架zxing解析
private void analyzeQRCode(@NonNull ImageProxy imageProxy) {
ByteBuffer byteBuffer = imageProxy.getPlanes()[0].getBuffer();
byte[] data = new byte[byteBuffer.remaining()];
byteBuffer.get(data);
...
BinaryBitmap bitmap = new BinaryBitmap(new HybridBinarizer(source));
Result result;
try {
result = multiFormatReader.decode(bitmap);
}
...
showQRCodeResult(result);
imageProxy.close();
}
private void showQRCodeResult(@Nullable Result result) {
if (binding != null && binding.qrCodeResult != null) {
binding.qrCodeResult.post(() ->
binding.qrCodeResult.setText(result != null ? "Link:\n" + result.getText() : ""));
binding.qrCodeResult.playSoundEffect(SoundEffectConstants.CLICK);
}
}
Video recording
Relying on the startRecording() of VideoCapture, video recording can be performed.
Add a button for switching between image shooting and video recording mode on the demo, and bind the UseCase of video shooting to the CameraProvider when switching to the video recording mode.
public void onVideoGo(View view) {
bindPreview(mCameraProvider, binding.previewView, isVideoMode);
}
private void bindPreview(@NonNull ProcessCameraProvider cameraProvider,
PreviewView previewView, boolean isVideo) {
...
mVideoCapture = new VideoCapture.Builder()
.setTargetRotation(previewView.getDisplay().getRotation())
.setVideoFrameRate(25)
.setBitRate(3 * 1024 * 1024)
.build();
cameraProvider.unbindAll();
if (isVideo) {
mCamera = cameraProvider.bindToLifecycle(this, cameraSelector,
mPreview, mVideoCapture);
} else {
mCamera = cameraProvider.bindToLifecycle(this, cameraSelector,
mPreview, mImageCapture, mImageAnalysis);
}
mPreview.setSurfaceProvider(previewView.getSurfaceProvider());
}
After clicking the record button, first make sure to obtain the external storage and audio permissions, and then start the video recording.
public void onCaptureGo(View view) {
if (isVideoMode) {
if (!isRecording) {
// Check permission first.
ensureAudioStoragePermission(REQUEST_STORAGE_VIDEO);
}
}
...
}
private void ensureAudioStoragePermission(int requestId) {
...
if (requestId == REQUEST_STORAGE_VIDEO) {
if (ActivityCompat.checkSelfPermission(this, Manifest.permission.WRITE_EXTERNAL_STORAGE)
!= PackageManager.PERMISSION_GRANTED
|| ActivityCompat.checkSelfPermission(this, Manifest.permission.RECORD_AUDIO)
!= PackageManager.PERMISSION_GRANTED) {
ActivityCompat.requestPermissions(...);
return;
}
recordVideo();
}
}
private void recordVideo() {
try {
mVideoCapture.startRecording(
new VideoCapture.OutputFileOptions.Builder(getContentResolver(),
MediaStore.Video.Media.EXTERNAL_CONTENT_URI, contentValues)
.build(),
CameraXExecutors.mainThreadExecutor(),
new VideoCapture.OnVideoSavedCallback() {
@Override
public void onVideoSaved(@NonNull VideoCapture.OutputFileResults outputFileResults) {
// Notify user...
}
}
);
}
...
toggleRecordingStatus();
}
private void toggleRecordingStatus() {
// Stop recording when toggle to false.
if (!isRecording && mVideoCapture != null) {
mVideoCapture.stopRecording();
}
}
Episode
A problem was found when implementing the video recording function.
When the video recording button is clicked, if the audio permission has not been obtained at the moment, the permission will be applied for. Even after obtaining permission to call the shooting interface, an exception will still occur. The log shows that the AudioRecorder instance is null and NPE is triggered.
Looking closely at the relevant logic, it is found that the current processing of the demo is to bind VideoCapture to the CameraProvider when switching to the video recording mode. At this point in time, if the audio permission has not been obtained, the AudioRecorder will not be initialized.
In fact, the corresponding prompt will be given in the log: AudioRecord object cannot initialized correctly. But why does NPE still occur when I get permission to call VideoCapture's shooting interface?
Because the internal processing of the shooting interface startRecording() is that if the AudioRecorder instance is null, the request will be terminated directly. No matter how many times it is called later, it will not help. In fact, there is logic in the latter part of the function to obtain the AudioRecorder instance again, but there is no chance to execute it because of the previous NPE.
// VideoCapture.java
public void startRecording(
@NonNull OutputFileOptions outputFileOptions, @NonNull Executor executor,
@NonNull OnVideoSavedCallback callback) {
...
try {
// mAudioRecorder为null将引发NPE终止录制的请求
mAudioRecorder.startRecording();
} catch (IllegalStateException e) {
postListener.onError(ERROR_ENCODER, "AudioRecorder start fail", e);
return;
}
...
mRecordingFuture.addListener(() -> {
...
if (getCamera() != null) {
// 前面发生了NPE,那么将失去此处再次获得AudioRecorder实例的机会
setupEncoder(getCameraId(), getAttachedSurfaceResolution());
notifyReset();
}
}, CameraXExecutors.mainThreadExecutor());
...
}
I don't know if this is a loophole in the implementation of VideoCapture or the developer deliberately. However, it seems unreasonable that NPE still occurs when the recording interface is called when the audio permission is clearly obtained.
There are only some evasive schemes that can be taken at the moment, or should developers do this?
Now the binding of VideoCapture is performed before the audio permission is obtained. This may cause the repeated NPE mentioned above. So change it to get audio permission and then bind VideoCapture to avoid it.
Having said that, would it be better to add instructions for obtaining audio permissions in the VideoCaptue documentation?
Camera effect extension
The use of the above-mentioned scenes alone cannot meet the ever-increasing demand for shooting. Camera effects such as portrait, night shooting, and beauty are essential. Fortunately, CameraX supports extended effects. But not all devices are compatible with this extension. You can check the device compatibility list on the official website for details.
The effects that can be extended are mainly divided into two categories.
One is PreviewExtender for effect extension during image preview, and the other is ImageCaptureExtender for effect extension during image capture.
Each category contains several typical effects.
NightPreviewExtender
BokehPreviewExtender portrait preview
BeautyPreviewExtender
HdrPreviewExtender HDR preview
AutoPreviewExtender automatic preview
The realization of turning on these effects is also very simple.
private void bindPreview(@NonNull ProcessCameraProvider cameraProvider,
PreviewView previewView, boolean isVideo) {
Preview.Builder previewBuilder = new Preview.Builder();
ImageCapture.Builder captureBuilder = new ImageCapture.Builder()
.setTargetRotation(previewView.getDisplay().getRotation());
...
setPreviewExtender(previewBuilder, cameraSelector);
mPreview = previewBuilder.build();
setCaptureExtender(captureBuilder, cameraSelector);
mImageCapture = captureBuilder.build();
...
}
private void setPreviewExtender(Preview.Builder builder, CameraSelector cameraSelector) {
BeautyPreviewExtender beautyPreviewExtender = BeautyPreviewExtender.create(builder);
if (beautyPreviewExtender.isExtensionAvailable(cameraSelector)) {
// Enable the extension if available.
beautyPreviewExtender.enableExtension(cameraSelector);
}
}
private void setCaptureExtender(ImageCapture.Builder builder, CameraSelector cameraSelector) {
NightImageCaptureExtender nightImageCaptureExtender = NightImageCaptureExtender.create(builder);
if (nightImageCaptureExtender.isExtensionAvailable(cameraSelector)) {
// Enable the extension if available.
nightImageCaptureExtender.enableExtension(cameraSelector);
}
}
Unfortunately, the Redmi 6A in my hand is not in the list of devices that support OEM effect expansion, so I can’t show you a sample image of the successful expansion effect.
High-level usage
In addition to the above-mentioned common camera usage scenarios, there are other optional configuration methods. The space limit will not be expanded in detail anymore, and those interested can refer to the official website to try it out.
Conversion output CameraX supports the conversion of image data and output, such as drawing a face block diagram after applying it to portrait recognition
https://developer.android.google.cn/training/camerax/transform-output?hl=zh-cn
Use case rotation The screen may rotate during image capture and analysis. Learn how to configure so that CameraX can obtain the screen orientation and rotation angle in real time to capture the correct image https://developer.android.google.cn/training/ camerax/orientation-rotation?hl=zh-cn
Configuration options Control resolution, auto focus, viewfinder shape settings and other configuration guidelines https://developer.android.google.cn/training/camerax/configuration?hl=zh-cn
Use attention
- Remember to call unbindAll() before calling bindToLifecycle() of CameraProvider, otherwise, an exception of repeated binding may occur.
- The analyze() of ImageAnalyzer should call the close() of ImageProxy to release the image immediately after analyzing the image, so that subsequent images can continue to be transmitted. Otherwise, the callback will be blocked. Therefore, we should also pay attention to the time-consuming problem of analyzing images.
- Do not store a reference to each ImageProxy instance after it is closed, because once close() is called, these images will become illegal.
- After the image analysis is over, you should call the clearAnalyzer() of ImageAnalysis to tell you not to stream the image to avoid performance waste.
- Do not forget to obtain audio permission for video recording scenes.
Interesting compatibility handling
When implementing the image capture function, I found such an interesting note in the takePicture() document of ImageCapture.
Before triggering the image capture pipeline, if the save location is a File or MediaStore, it is first verified to ensure it’s valid and writable.
A File is verified by attempting to open a FileOutputStream to it, whereas a location in MediaStore is validated by ContentResolver#insert() creating a new row in the user defined table,
retrieving a Uri pointing to it, then attempting to open an OutputStream to it.
The newly created row is ContentResolver#delete() deleted at the end of the verification.
On Huawei devices, this deletion results in the system displaying a notification informing the user that a photo has been deleted.
In order to avoid this, validating the image capture save location in MediaStore is skipped on Huawei devices.
The main idea is that if the saved Uri is a MediaStore, a line will be inserted to verify whether the save path is legal and writable. The test line will be deleted after verification.
However, deleting the line record on the Huawei device will trigger a notification to delete the photo. Therefore, in order to avoid bothering users, CameraX will skip the path verification on Huawei devices.
class ImageSaveLocationValidator {
// 将判断设备品牌是否为华为或荣耀，是则直接跳过验证
static boolean isValid(final @NonNull ImageCapture.OutputFileOptions outputFileOptions) {
...
if (isSaveToMediaStore(outputFileOptions)) {
// Skip verification on Huawei devices
final HuaweiMediaStoreLocationValidationQuirk huaweiQuirk =
DeviceQuirks.get(HuaweiMediaStoreLocationValidationQuirk.class);
if (huaweiQuirk != null) {
return huaweiQuirk.canSaveToMediaStore();
}
return canSaveToMediaStore(outputFileOptions.getContentResolver(),
outputFileOptions.getSaveCollection(), outputFileOptions.getContentValues());
}
return true;
}
...
}
public class HuaweiMediaStoreLocationValidationQuirk implements Quirk {
static boolean load() {
return "HUAWEI".equals(Build.BRAND.toUpperCase())
|| "HONOR".equals(Build.BRAND.toUpperCase());
}
/**
* Always skip checking if the image capture save destination in
* {@link android.provider.MediaStore} is valid.
*/
public boolean canSaveToMediaStore() {
return true;
}
}
Advantages of CameraX
It comes from the fact that CameraX has carried out a high degree of encapsulation on the basis of Camera2 and has carried out compatibility processing for a large number of devices, which makes CameraX have many advantages.
- Ease of use The encapsulated API can achieve the goal efficiently.
- Equipment consistency Don't care about the version, ignore hardware differences, and achieve a consistent development experience.
- New camera experience The same beauty and other shooting functions as the original camera can be realized through the effect extension.
Conclusion
CameraX was released on August 7, 2019. It has been updated from the alpha version to the current beta version.
From the above interesting Huawei device compatibility processing, we can see the determination of CameraX to dominate the rivers and lakes.
The latest is still a beta version and needs to be improved, but it is not impossible to put it into a production environment.
With such a useful framework, everyone should use it more and give suggestions so that it can become more and more perfect, and can bring good news to developers and users.