How does the access side of GB28181 equipment support H.265?

technical background

For a long time, the specification version before GB28181-2022 has been criticized: there is no clear description for H.265, especially for surveillance cameras. With a clear explanation, let's interpret the specification together:

PS encapsulation of video and audio data based on RTP

RTP-based PS encapsulation first encapsulates the video and audio streams into PS packets according to ISO/IEC 13818-1:2019, and then encapsulates the PS packets into RTP packets in the form of payload.

When performing PS encapsulation, you should put the team Map), the system header and the PSM after the PS header, the first ti (System Header) and PSM (Program Stream Map), the system header and the PSM after the PS header, Before the first PES pack.

A typical video key frame PS packet structure is shown in Figure C.1, where PESV is a video PES packet, PESA is an audio PES packet, and the PS packet structure of a video non-key frame generally does not include the system header and PSM. For the specific data structure of each part in the PS package, refer to the relevant description in ISO/IEC13818-1:2019.

The system header should contain a description of the stream type in the PS package, where the video and audio stream ID (stream_id) values ​​are as follows: Video stream ID: 0xEO;

For several video and audio formats specified in this document, the values ​​of the stream type (stream_type) in PSM are as follows: H.265 video stream: 0x24;

RTP-based video and audio encapsulation

RTP encapsulation of H.265 video stream

The RTP payload format of H.265 shall conform to the relevant regulations of IETF RFC 7798.

Selection of the payload type (Payload Type) identification number of the H.265 video stream RTP packet: select from the dynamic range (96~~127) in Table 5 of the IETF RFC3551 protocol, it is recommended to set it as 100, and fill it according to actual needs.

H.265 Video Coding and Decoding Technical Requirements

Grades and levels of H.265

Video coding using the H.265 standard should at least support the main profile (MainProfile) of the ITU-T H.265 (2019) video standard. The application should extend support to Level 4; the level and level supported by video decoding should not be lower than the highest level and level supported by encoding, and should at least support Level 4 of the main level of the H.265 video standard. Video decoding should extend support for B-frame tools in the H.265 Main Profile, and the number of B-frames between two adjacent Р-frames should not exceed 2.

For a detailed description of the H.265 main profile video coding standard, see the relevant provisions of ITU-T H.265 (2019).

H.265 Main Profile Options and Tools

The options and tools supported by H.265 main profile mainly include:

a) The bit depth is limited to 8 bits;

b) The sampling limit is 4:2:0;

c) CTB size from 16×16 to 64×64;

d) Under the maximum resolution supported by Level, the cache capacity of decoded images is limited to 6 images.

The resolution becomes smaller, and the buffer capacity of the decoded image can be greater than 6 images, but should not exceed 16 images;

e) The wavefront and slice division methods are allowed to be selected, but cannot be selected at the same time.

The video stream adopting the H.265 coding standard shall be the H.265 main profile video stream, and the encoding shall support some or all of the above main profile options and tools; the H.265 decoding shall at least support all the above options and tools.

When encoding multiple reference frames, the number of reference frames of a P slice is generally not more than two frames, and should not exceed 15 frames.

In order to ensure the efficiency of code stream parsing, the corresponding video parameter set (Video Param-eter Set, VPS), sequence parameter set (Sequence Parameter Set, SPS) and image parameter set ( Picture Parameter Set, PPS).

Technical realization

In fact, when we implemented GB28181-2016, we had already made settings for H.265:

The settings are as follows:

//视频编码类型选择++++++++++
        videoEncodeTypeSelector = (Spinner)findViewById(R.id.videoEncodeTypeSelector);
        final String[] videoEncodeTypes = new String[]{"软编(H.264)", "硬编(H.264)", "硬编(H.265)"};
        ArrayAdapter<String> adapterVideoEncodeType = new ArrayAdapter<String>(this,
                android.R.layout.simple_spinner_item, videoEncodeTypes);
        adapterVideoEncodeType.setDropDownViewResource(android.R.layout.simple_spinner_dropdown_item);
        videoEncodeTypeSelector.setAdapter(adapterVideoEncodeType);

        videoEncodeTypeSelector.setOnItemSelectedListener(new AdapterView.OnItemSelectedListener() {

            @Override
            public void onItemSelected(AdapterView<?> parent, View view,
                                       int position, long id) {

                if (isRTSPPublisherRunning || isPushingRtmp || isGB28181StreamRunning || isRecording) {
                    Log.e(TAG, "Could not switch video encoder type during publishing..");
                    return;
                }

                videoEncodeType = position;
                Log.i(TAG, "[视频编码类型]Currently choosing: " + videoEncodeTypes[position] + ", videoEncodeType: " + videoEncodeType);
            }

            @Override
            public void onNothingSelected(AdapterView<?> parent) {

            }
        });

Considering the encoding performance of Android devices, we only support H.265 encoding on the Android platform. If you need H.265 hard encoding, you only need to call the SetSmartPublisherVideoHevcHWEncoder() interface. The second parameter is for the encoding rate settings.

/**
     * Author: daniusdk.com
     * Set Video H.265(hevc) hardware encoder, if support H.265(hevc) hardware encoder, it will return 0(设置H.265硬编码)
     *
     * @param kbps: the kbps of different resolution.
     *
     * @return {0} if successful
     */
    public native int SetSmartPublisherVideoHevcHWEncoder(long handle, int kbps);

Of course, Native Media NDK hard coding can also be used. It should be noted that Native Media NDK hard coding requires an Android 5.0 or higher system, and the interface is not as complete as the native one:

    /**
     * 设置视频硬编码是否使用 Native Media NDK, 默认是不使用, 安卓5.0以下设备不支持
     * @param handle
     * @param is_native: 0表示不使用, 1表示使用, sdk默认是0.
     * @return {0} if successful
     */
    public native int SetNativeMediaNDK(long handle, int is_native);

For H.265 hard coding, other interfaces are designed as follows:

	/*
	* 设置视频硬编码码率控制模式
	* @param hw_bitrate_mode: -1表示使用默认值, 不设置也会使用默认值, 0:CQ, 1:VBR, 2:CBR, 3:CBR_FD, 请参考:android.media.MediaCodecInfo.EncoderCapabilities
	* 注意硬编码和手机硬件有关，多数手机只支持部分码率模式, 另外硬编码设备差异很大，不同设备同一码率控制模式效果可能不一样
	* @return {0} if successful
	*/
	public native int SetVideoHWEncoderBitrateMode(long handle, int hw_bitrate_mode);


	/*
	 * 设置视频硬编码复杂度, 安卓5.0及以上支持
	 * @param hw_complexity: -1表示不设置, 请参考:android.media.MediaCodecInfo.EncoderCapabilities.getComplexityRange() 和 android.media.MediaFormat.KEY_COMPLEXITY
	 * 注意硬编码和手机硬件有关，部分手机可能不支持此设置
	 * @return {0} if successful
	 */
	public native int SetVideoHWEncoderComplexity(long handle, int hw_complexity);

	/*
	 * 设置视频硬编码质量, 安卓9及以上支持, 仅当硬编码器码率控制模式(BitrateMode)是CQ(constant-quality mode)时才有效
	 * @param hw_quality: -1表示不设置, 请参考:android.media.MediaCodecInfo.EncoderCapabilities.getQualityRange() 和 android.media.MediaFormat.KEY_QUALITY
	 * 注意硬编码和手机硬件有关，部分手机可能不支持此设置
	 * @return {0} if successful
	 */
	public native int SetVideoHWEncoderQuality(long handle, int hw_quality);

	/*
	 * 设置视频硬编码最大码率, 安卓没有相关文档说明, 所以不建议设置,
	 * @param hw_max_bitrate: 每秒最大码率, 单位bps
	 * @return {0} if successful
	 */
	public native int SetVideoHWEncoderMaxBitrate(long handle, long hw_max_bitrate);

The relevant calls are as follows:

int hevcHWKbps = setHardwareEncoderKbps(false, video_width_, video_height_);
hevcHWKbps = hevcHWKbps*fps/25;
Log.i(TAG, "hevcHWKbps: " + hevcHWKbps);

int isSupportHevcHWEncoder = libPublisher
                    .SetSmartPublisherVideoHevcHWEncoder(publisherHandle, hevcHWKbps);

if (isSupportHevcHWEncoder == 0) {
  libPublisher.SetNativeMediaNDK(publisherHandle, 0);
  libPublisher.SetVideoHWEncoderBitrateMode(publisherHandle, 0); // 0:CQ, 1:VBR, 2:CBR
  libPublisher.SetVideoHWEncoderQuality(publisherHandle, 39);

  // libPublisher.SetVideoHWEncoderMaxBitrate(publisherHandle, ((long)hevcHWKbps)*1200);

  Log.i(TAG, "Great, it supports hevc hardware encoder!");
}

Summarize

The description of GB28181-2022 for H.265 is a supplement to 2016. With the implementation and promotion of 2022 starting in July this year, it is believed that GB28181 will be applied to all walks of life, such as law enforcement recorders, smart helmets, smart monitoring, smart Scenarios such as retail, smart education, remote office, bright kitchen, smart transportation, smart construction site, Xueliang project, safe village, production and transportation, vehicle-mounted terminal, etc. It is not difficult for the access terminal of GB28181 equipment to support H.265. If the platform side needs web playback, generally speaking, it needs to be converted to H.264 for playback, which raises the requirements for the special performance of the national standard platform side. If it is a cs architecture, the problem is not big.