H264 encoding format

The function of H.264 is divided into two layers, the video coding layer VCL and the network abstraction layer NAL. The VCL data is an output after encoding processing , and represents a sequence of compressed and encoded video data . When VCL data is transmitted or stored, these encoded VCL data will be mapped or encapsulated into NAL units .

VCL, Video Coding Layer, video coding layer, pure video data content information after encoding, without any redundant header information;
NAL, Network Abstraction Layer, network abstraction layer, is responsible for packaging and sending in an appropriate way during network transmission ;

To put it simply, during video data transmission, VCL data will be packaged into NAL basic units to transmit NAL unit data .

insert image description here

NAL

H.264The main goals of :

High video compression ratio;
Good network affinity, it can be applied to various transmission networks;

NAL, Network Abstraction Layer, is defined as follows:
:::info
The Network Abstraction Layer is a part of the H.264/ANC and HEVC video coding standards. (Video coding standard)
The main goal of the NAL is the provision (supply) of a “network -friendly” video representation addressing “conversational (two-way)” (video telephony: video telephony) and “non conversational” (storage, broadcast, or streaming) applications. NAL has achieved a significant improvement in application flexibility relative to prior video coding
standards .
The main goal of NAL is to provide a network-friendly network-friendly video representation, addressing two-way (video telephony) and one-way (non-video telephony - storage, broadcast or video playback) applications.
Compared with previous video coding standards, NAL has achieved significant improvement in application flexibility.
:::
Therefore, NALit is to package VCLdata for better transmission on the network.

NAL units

The encoded video data is packed into NAL units, each of which NAL unitsis a packet containing an integer number of bytes. NAL unitThe first byte of each is header, which contains the data type information. NAL unitThe structure definition is oriented packetand bitstreamdesigned, with good network transmission performance.

In the actual H264 data frame, there is often (NAL unit)a separator 00 00 00 01 or 00 00 01delimiter between the frames StartCode, which is called NALUthe beginning of a frame. Therefore, an original NALUstructure contains three parts:

StartCode
NALU Header
NALU Payload (byte sequence load, RBSP, Raw Byte Sequence Payload)

NALU Header
NALU Header is a byte immediately after StartCode , and the meaning of each bit is as follows:
F: forbidden, forbidden bit, occupying the first bit of the NAL header, when the forbidden bit value is 1, it means a syntax error;
NRI: Reference level, occupying the second and third bits of the NAL header, the larger the value, the more important the NAL unit is;
Type: NAL unit type, which occupies the fourth to eighth digits of the NAL header. Common NAL unit types are as follows:

NAL unitcan be classified into VCLand non-VCL NAL units. Which VCL NAL unitcontains the data frame of the video image, non-VCL NAL unitwhich contains additional information, such as SPSand PPSparameter set information (important header data that can apply to a large number of VCL NAL units)and SEIsupplementary enhancement information (timing information and other supplemental data that may enhance usability of the decoded video signal but are not necessary for decoding the values of the samples in the video pictures).
IDR frames. An IDR frame is an I frame, but an I frame is not necessarily an IDR frame. In a complete video stream unit, the first image frame is an IDR frame, and the IDR frame is a forced refresh frame. During the decoding process, when an IDR frame appears, the sps and pps must be updated to prevent errors in the previous I frame. Cause sps, pps refer to I-frames lead to uncorrectable.
SEI, Supplemental Enhancement Information, provides a method to add additional information to the video stream;
SPS, Sequence Parameter Set, a set of global parameters of a coded video sequence (Coded Video Sequence) is stored in the SPS. So this type saves the parameters related to the encoding sequence.
PPS, Picture Paramater Set, this type saves the parameters related to the overall image.
AU delimiter, the full name of AU is Access Unit, which is a collection of one or more NALUs, representing a complete frame.

Parse H264 data

According to the above H264encoding format analysis, it is possible to startcodedistinguish each NAL unit, and then analyze NAL unitthe first byte of each to obtain NAL unitthe importance NAL Typeand length information. The specific code is as follows:

#include <stdio.h>
#include <stdlib.h>
#include <string.h>

typedef enum {
    
    
    NALU_TYPE_SLICE = 1,    // 不分区，非IDR图像
    NALU_TYPE_DPA,          // 片分区A
    NALU_TYPE_DPB,          // 片分区B
    NALU_TYPE_DPC,          // 片分区C
    NALU_TYPE_IDR,          // IDR图像中的片
    NALU_TYPE_SEI,         // 补充增强信息单元
    NALU_TYPE_SPS,          // 序列参数集
    NALU_TYPE_PPS,          // 图像参数集
    NALU_TYPE_AUD,          // 分界符
    NALU_TYPE_EOSEQ,        // 序列结束符
    NALU_TYPE_EOSTREAM,     // 码流结束符
    NALU_TYPE_FILL          // 填充
} NaluType;

typedef enum {
    
    
    NALU_PRIORITY_DISPOSABLE = 0,
    NALU_PRIORITY_LOW,
    NALU_PRIORITY_HIGH,
    NALU_PRIORITY_HIGHEST
} NaluPriority;


typedef struct {
    
    
    int pos;        // NALU对应起始码，在文件中的位置
    int codelen;    // 起始码大小
    int header;     // (0) + (priority)2bits + (types)5bits
} Nalu_t;

void fill_typestr(char *type_str, int type) {
    
    
    switch(type) {
    
    
        case NALU_TYPE_SLICE:sprintf(type_str,"SLICE");break;
        case NALU_TYPE_DPA:sprintf(type_str,"DPA");break;
        case NALU_TYPE_DPB:sprintf(type_str,"DPB");break;
        case NALU_TYPE_DPC:sprintf(type_str,"DPC");break;
        case NALU_TYPE_IDR:sprintf(type_str,"IDR");break;
        case NALU_TYPE_SEI:sprintf(type_str,"SEI");break;
        case NALU_TYPE_SPS:sprintf(type_str,"SPS");break;
        case NALU_TYPE_PPS:sprintf(type_str,"PPS");break;
        case NALU_TYPE_AUD:sprintf(type_str,"AUD");break;
        case NALU_TYPE_EOSEQ:sprintf(type_str,"EOSEQ");break;
        case NALU_TYPE_EOSTREAM:sprintf(type_str,"EOSTREAM");break;
        case NALU_TYPE_FILL:sprintf(type_str,"FILL");break;
    }
}

void fill_priostr(char *priostr, int prio) {
    
    
    switch(prio) {
    
    
        case NALU_PRIORITY_DISPOSABLE:sprintf(priostr,"DISPOS");break;
        case NALU_PRIORITY_LOW:sprintf(priostr,"LOW");break;
        case NALU_PRIORITY_HIGH:sprintf(priostr,"HIGH");break;
        case NALU_PRIORITY_HIGHEST:sprintf(priostr,"HIGHEST");break;
    }
}


int startcode_len(unsigned char *buf) {
    
    
    if (buf[0] == 0 && buf[1] == 0) {
    
    
        if (buf[2] == 1) return 3;
        else if(buf[3] == 1) return 4;
    }    
    return -1;
}

void h264_analyse(const char *file) {
    
    
    FILE *fp = fopen(file, "r");
    unsigned char *buf = malloc(30000000); // 根据视频大小设
    Nalu_t *nalu = malloc(sizeof(*nalu));
    memset(nalu, 0, sizeof(Nalu_t));
    
    // 获取文件大小
    fseek(fp, 0, SEEK_END);
    int filesize = ftell(fp);
    rewind(fp);

    // 先读部分数据，解析第一个NALU
    fread(buf, 1, 100, fp);
    nalu->pos = 0;
    nalu->codelen = startcode_len(&buf[nalu->pos]);

    printf("| NUM | CODE |   PRIO |   TYPE  | LEN |\n");

    int cnt = 0;                // nalu个数
    int left = 0, right = 100;  // left表示当前搜索位置，right表示当前文件偏移
    int found, len;             
    while (1) {
    
    
        int headidx = nalu->pos + nalu->codelen;  // NALU header
        nalu->header = buf[headidx];
        // int nForbiddenBit = (nFrameType >> 7) & 0x1;//第1位禁止位，值为1表示语法出错
        // int nReference_idc = (nFrameType >> 5) & 0x03;//第2~3位为参考级别
        // int nType = nFrameType & 0x1f;//第4~8为是nal单元类型
        int type = nalu->header & 0x1F;             // NALU Type
        char type_str[10] = {
    
    0};
        fill_typestr(type_str, type);

        // int prio = (nalu->header & 0x60)>>5;
        int prio = (nalu->header >> 5) & 0x03;
        char prio_str[10] = {
    
    0};
        fill_priostr(prio_str, prio);
        
        // 找到下一个起始码
        found = 0; len = 0;
        left += nalu->codelen + 1;  // 跳过startcode和header
        while (!found) {
    
    
            if (left < right) {
    
    
                if (left + 4 >= filesize) // 防止在函数内数组越界
                    goto clean;
                len = startcode_len(&buf[left++]);
                if (len > 0) found = 1;
            } else {
    
    
                if (right + 100 < filesize) {
    
    
                    fread(&buf[right], 1, 100, fp);
                    right += 100;
                } else {
    
        // 剩余的数据不到100B字节
                    fread(&buf[right], 1, filesize - right, fp);
                    right = filesize;
                }
            }
        }
        int nalulen = left - nalu->pos - nalu->codelen;  // NALU大小，不包括起始码

        printf("|%5d|%6d|%8s|%9s|%5d|\n", cnt, nalu->codelen, prio_str, type_str, nalulen);
        cnt++;
        nalu->codelen = len;
        nalu->pos = left - 1;
    }

clean:
    free(nalu);
    free(buf);
    fclose(fp);
}

int main(int argc, char const* argv[])
{
    
    
    h264_analyse("/opt/nvidia/deepstream/deepstream-6.1/samples/streams/sample_720p.h264"); 
    return 0;
}

The file parses the information of each NAL unit as follows:

| NUM | CODE |   PRIO |   TYPE  | LEN |
|    0|     4|  DISPOS|      AUD|    3|
|    1|     4| HIGHEST|      SPS|   26|
|    2|     4| HIGHEST|      PPS|    6|
|    3|     4|  DISPOS|      SEI|  704|
|    4|     4| HIGHEST|      IDR|47583|
|    5|     4|  DISPOS|      AUD|    3|
|    6|     4|    HIGH|    SLICE|25063|
|    7|     4|  DISPOS|      AUD|    3|
|    8|     4|  DISPOS|    SLICE| 8783|
|    9|     4|  DISPOS|      AUD|    3|
|   10|     4|    HIGH|    SLICE| 3004|
|   11|     4|     LOW|         |23386|
|   12|     4|  DISPOS|      AUD|    3|
|   13|     4|  DISPOS|    SLICE| 7633|
...

H264 Analysis Tool Elecard StreamEye Tools

The Elecard StreamEye Tools is a set of powerful tools designed for professionals and prosumers in the video compression field. The Elecard StreamEye Tools enables the user to perform an effective in-depth analysis of video sequences. The Elecard StreamEye Tools is a set of powerful tools
, Designed for professionals and prosumers in the field of video compression. Elecard StreamEye Tools enables users to perform effective in-depth analysis of video sequences.

Link: https://pan.baidu.com/s/1MUy6rs3MuDW6PUGkhDHy6g?pwd=of07 Extraction code: of07

The Elecard StreamEye Tools toolkit contains the following five applications:
Elecard StreamEye
Elecard YUV Viewer
Elecard Video QuEst (Quality Estimator)
Elecard Stream Analyzer
Elecard Buffer Analyzer

insert image description here

Elecard Stream Analyzer

Elecard Stream Analyzer is a powerful tool designed for syntax analysis (grammar analysis) of encoded media streams (encoded media data) and presentation of the analysis log in a human readable form. Stream Analyzer operates with MPEG-1 Video/Audio, MPEG- 2 Video/Audio, AAC, AC-3 and AVC/H.264 files (supported data types).

H264Analysis NAL unit_

By comparing with the results of the above code analysis, it can be seen that the above analysis code is correct.

H264file HEXviewer

Elecard StreamEye

The StreamEye provides the user with a visual representation of the encoded video features and a stream structure analysis of MPEG-1/2/4 or AVC/Н.264 Video Elementary Streams (VES), MPEG-1 System Streams (SS), MPEG-2 Program Streams (PS) and MPEG-2 Transport Streams (TS).

insert image description here

Overview of some functions:

bitrateanalyze

Selected bitratestate, display the corresponding bitratecurve

color space
file information
The motion vector

StreamEye tool is very powerful and can be analyzed according to your needs.

Analysis of H264