The video recording error reported by the Android car machine is actually the fault of the encoder?

1. Description of the phenomenon

There is a project at work that uses CameraView , a third-party camera library, to call the camera.
CameraViewEncapsulated Camera1and Camera2, many functions are encapsulated inside, and APIit is relatively simple to use.
After connecting in Appthe middle, the video can be recorded normally on the mobile phone, and there seems to be no problem. But when I installed it on the car
with great joy , I found that the error shown in the picture below will be reported when recording a video on the car:Appjava.lang.illegalStateException: at MediaCodec.native_dequeueOutputBuffer

insert image description here

Why is this? The reason for this problem will be explored below.

2. MediaCodec related API description

First, let's introduce the relevantMediaCodec API

2.1 Get all supported encoders

MediaCodecInfo[] array = new MediaCodecList(MediaCodecList.REGULAR_CODECS).getCodecInfos();

The list of encoders obtained on the car, you can see that the supported video/avchard codes are OMX.qcom.video.encoder.avc, and the soft codes areOMX.google.h264.encoder

codecName:OMX.google.aac.encoder supportedTypes:audio/mp4a-latm
codecName:OMX.google.amrnb.encoder supportedTypes:audio/3gpp
codecName:OMX.google.amrwb.encoder supportedTypes:audio/amr-wb
codecName:OMX.google.flac.encoder supportedTypes:audio/flac
codecName:OMX.qcom.video.encoder.avc supportedTypes:video/avc
codecName:OMX.google.h264.encoder supportedTypes:video/avc
codecName:OMX.qcom.video.encoder.h263sw supportedTypes:video/3gpp
codecName:OMX.google.h263.encoder supportedTypes:video/3gpp
codecName:OMX.qcom.video.encoder.hevc supportedTypes:video/hevc
codecName:OMX.qcom.video.encoder.mpeg4sw supportedTypes:video/mp4v-es
codecName:OMX.google.mpeg4.encoder supportedTypes:video/mp4v-es
codecName:OMX.qcom.video.encoder.vp8 supportedTypes:video/x-vnd.on2.vp8
codecName:OMX.google.vp8.encoder supportedTypes:video/x-vnd.on2.vp8
codecName:OMX.google.vp9.encoder supportedTypes:video/x-vnd.on2.vp9

MediaCodecList.REGULAR_CODECSIndicates that only standard, stable codecs are fetched. Using this enumeration value can ensure that the obtained codecs are sorted according to the standard and stability of the Android platform, which can avoid pitfalls in audio and video development to a certain extent.

MediaCodecList.ALL_CODECSMeans to get all available codecs, including possibly non-standard, unstable codecs. Using this enumeration value can get more codec information, but there may be some unstable factors.

2.2 Create MediaCodec by specifying an encoder

In this way, a MediaCodec with a specified encoder will be created

MediaCodec codec = MediaCodec.createEncoderByType("OMX.google.h264.encoder")

2.3 Create MediaCodec by specifying Mine

mimeTypeIn this way, an encoder that is most recommended by the system under that category is created .

MediaCodec codec = MediaCodec.createEncoderByType("video/avc");

3. Which encoder to choose?

At this point, you may be wondering, which one should be selected with so many encoders?

First of all miniType, we video/avcjust need to choose video recording, that is h264.

video/avcThe encoder in the car machine has OMX.qcom.video.encoder.avcand OMX.google.h264.encoder.

Here, let's learn about hard coding and soft coding.

3.1 Hardcoding and softcoding

Soft decoding is to use the computing power of the CPU to decode. If the CPU is not very strong, the decoding speed will be slower, and the phone will also heat up. However, due to the use of a unified algorithm, the compatibility will be very good.
Hard decoding is to use the special decoding chip on the mobile phone to speed up the decoding. Generally, the speed of hard decoding is much faster, but because hard decoding is implemented by various manufacturers, the quality is uneven, and compatibility problems are prone to occur.

	advantage	shortcoming
soft coding	1. Strong compatibility, low requirements on the system version, and less errors. 2. In terms of decoding, the color of soft decoding is generally softer than that of hard decoding. 3. The encoding space is relatively large and the degree of freedom is high.	1. High CPU consumption 2. The machine tends to heat up 3. High power consumption
hard decoding	Low power consumption, high execution efficiency	1. Different types of chips implement codecs differently, and there is no guarantee that the decoding effect is the same as other models or error-free. 2. Poor controllability, relying on the underlying codec implementation 3. Difficult to upgrade and maintain

3.2 How to judge hard coding and soft coding

In general, the encoders omx.google.at the beginning and the beginning are soft encodings.c2.android.

In addition, all encoders that do not start with and can also be classified as soft encoding omx..c2.

Others can be considered as implicit coding, and the specific judgment code is as follows.

boolean isHardwareEncoder(@NonNull String encoder) {
    
    
    encoder = encoder.toLowerCase();
    boolean isSoftwareEncoder = encoder.startsWith("omx.google.")
        || encoder.startsWith("c2.android.")
        || (!encoder.startsWith("omx.") && !encoder.startsWith("c2."));
    return !isSoftwareEncoder;
}

3.3 Solve recording crash

At this point, we know that we can use soft decoders, that is, OMX.google.h264.encoderto reduce compatibility issues.

Indeed, we tried to use OMX.google.h264.encoderthis soft coding, and the video can be recorded normally on the car.

But what exactly is this for?

4. Third-party library CameraView source code analysis

Next, let's explore CameraViewthe source code. The following CameraViewsource code analysis is based on CameraView 2.7.2the version

4.1 Initialize MediaCodec

Let's first look at MediaCodecwhen it was initialized.

You can find the initialization method MediaCodecin VideoMediaEncoder.javathis classonPrepare

This method will create exactly one encoder if you pass create mConfig.encoderwithout it .NULLcreateByCodecNameMediaCodec

Otherwise, through createEncoderByTypecreation , the system will create a most recommended one MediaCodecbased on the incoming one .mineTypeMediaCodec

if (mConfig.encoder != null) {
    
    
    mMediaCodec = MediaCodec.createByCodecName(mConfig.encoder);
} else {
    
    
    mMediaCodec = MediaCodec.createEncoderByType(mConfig.mimeType); //mineType默认为video/avc
}

Let's take a look at mConfig.encoderwhere it is assigned

SnapshotVideoRecorder#onRendererFrame()There is such a line in the trace

videoConfig.encoder = deviceEncoders.getVideoEncoder();

instead getVideoEncoderof callingmVideoEncoder.getName()

public String getVideoEncoder() {
    
    
    if (mVideoEncoder != null) {
    
    
        	return mVideoEncoder.getName();
        } else {
    
    
        	return null;
    }
}

4.2 When is mVideoEncoder assigned

Where did this mVideoEncodercome from?

It can be found that it is assigned DeviceEncodersin the constructormVideoEncoder

List<MediaCodecInfo> encoders = getDeviceEncoders();
mVideoEncoder = findDeviceEncoder(encoders, videoType, mode, videoOffset);

4.3 findDeviceEncoder parameter analysis

Take a look at parameter one encoders, which is getDeviceEncodersused to obtain all supported encoders

//调用MediaCodecList.getCodecInfos()，获取所有支持的编码器
List<MediaCodecInfo> getDeviceEncoders() {
    
    
    ArrayList<MediaCodecInfo> results = new ArrayList<>();
    MediaCodecInfo[] array = new MediaCodecList(MediaCodecList.REGULAR_CODECS).getCodecInfos();
    for (MediaCodecInfo info : array) {
    
    
        if (info.isEncoder()) results.add(info);
    }
    return results;
}

The second parameter videoTypeis by defaultvideo/avc

Parameter three has two options

MODE_RESPECT_ORDER: Look at the literal meaning, respect the order provided by the system
MODE_PREFER_HARDWARE: Literally, it is more inclined to hardcode

Parameter four videoOffset, by default 0, only the outer layer try-catchis abnormal, videoOffset++try the next decoder.

4.4 Internal implementation of findDeviceEncoder

Next, let's look at findDeviceEncoderthe encoder used to select the final

It will be matched first nimeType, and the same will be added to the waiting list
If modeyes mode == MODE_PREFER_HARDWARE, the candidate list will be sorted according to the rules of hard-coded priority
videoOffsetFinally, take out the corresponding encoder from the list to be selected according to this index

//选定最终的编码器
MediaCodecInfo findDeviceEncoder(@NonNull List<MediaCodecInfo> encoders,
                                 @NonNull String mimeType,
                                 int mode,
                                 int offset) {
    
    
    ArrayList<MediaCodecInfo> results = new ArrayList<>();
    for (MediaCodecInfo encoder : encoders) {
    
    
        String[] types = encoder.getSupportedTypes();
        for (String type : types) {
    
    
            if (type.equalsIgnoreCase(mimeType)) {
    
    
                results.add(encoder);
                break;
            }
        }
    }
    LOG.i("findDeviceEncoder -", "type:", mimeType, "encoders:", results.size());
    if (mode == MODE_PREFER_HARDWARE) {
    
    
        Collections.sort(results, new Comparator<MediaCodecInfo>() {
    
    
            @Override
            public int compare(MediaCodecInfo o1, MediaCodecInfo o2) {
    
    
                boolean hw1 = isHardwareEncoder(o1.getName());
                boolean hw2 = isHardwareEncoder(o2.getName());
                return Boolean.compare(hw2, hw1);
            }
        });
    }
    if (results.size() < offset + 1) {
    
    
        // This should not be a VideoException or AudioException - we want the process
        // to crash here.
        throw new RuntimeException("No encoders for type:" + mimeType);
    }
    return results.get(offset);
}

At this point, the acquisition order of the parser is clear, so VideoMediaEncoderwhen is it initialized?

4.5 When is VideoMediaEncoder initialized

There are two places to be called

SnapshotVideoRecorderConstructor -> The specific calling method is:takeVideo -> onTakeVideo -> 创建SnapshotVideoRecorder并调用其start()
FullVideoRecorderConstructor -> The specific calling method is:takeVideoSnapshot -> onTakeVideoSnapshot -> 创建FullVideoRecorder并调用其start()

4.6 Bugs in CameraView

SnapshotVideoRecorderCreated twice in , DeviceEncodersand the final effective one is DeviceEncoders.MODE_PREFER_HARDWARE(preferably hard-coded) DeviceEncoders, which causes the call takeVideoSnapshot(record video with filter) to use hard-coded first.

However, FullVideoRecorderit is only created once in DeviceEncoders, and the final effect is DeviceEncoders.MODE_RESPECT_ORDER(according to the order provided by the system), by takeVideocalling (recording video without filter).

insert image description here

At the same time, due to OMX.qcom.video.encoder.avcthe problem of this hard coding, there is also a problem with the function of selecting the appropriate video resolution. The selected resolution is width=192px,height=108px, but the actual one should be 1920*1080.

insert image description here

5. Summary

At this point, we understand why an error is reported directly when recording a video on the car.

Because the car machine system itself does not support OMX.qcom.video.encoder.avcthis hard coding, but it is returned in the support list OMX.qcom.video.encoder.avc, and it is placed in video/avcthe highest priority sequence. This is the most important and fundamental reason.

Secondly, when CameraViewthis library is recording a video with a filter ( ), because it was created twice , the hard code was finally selected , which led to a direct crash.takeVideoSnapshotBUGDeviceEncodersOMX.qcom.video.encoder.avcApp

When recording a video without a filter ( takeVideo), it is not created twice DeviceEncoders, BUGbut because video/avcthe highest priority sequence provided by the system is OMX.qcom.video.encoder.avc, it Appstill crashes.