1. The relationship between VCL and NAL:
1. The functions of H.264 are divided into two layers:
VCL (Video Codeing Layer): Video coding layer, which is responsible for the processing of video content, focusing on the coding and decoding algorithms;
NAL (Network Abstraction Layer) : Network abstraction layer, responsible for packaging and transmitting the encoded data in the format required by the network;
2. Three different data formats in the H.264 encoding process:
(1) SODB (String of Data Bits) data bit string: the
most original encoded data, namely VCL data, without any additional data;
(2) RBSP (Raw Byte Sequence Payload) raw byte sequence payload:
after SODB data is added a trailing bit (rbsp_trailing_bits), a bit of "1", several bits of "0", the purpose is for byte alignment;
RBSP = SODB + rbsp_trailing_bits
(3) EBSP (Encapsulated Byte Sequence Packets) extended byte sequence payload:
a byte "0x03" is added to prevent contention on the basis of RBSP data; the
reason is:
the start code of NALU in H264 is 0x000001 or 0x00000001,
and at the same time H264 stipulates that when 0x000000 is detected, it can also indicate the end of the current NALU,
but what if 0x000001 or 0x00000001 appears in the NALU?
Therefore, H264 proposes a mechanism of "preventing competition". When the encoder completes a NALU,
it should detect whether the following left data appears in the NALU. If they are detected, the
encoder will be at the last word. preganglionic, insert a new byte "0x03":
0x000000 -> 0x00000300
0x000001 -> 0x00000301
0x000002 -> 0x00000302
0x000003 -> 0x00000303
Note:
NALUs head is 0x000001 or 0x00000001, are processed in the above manner,
for 0x000001 is to insert 0x03 after the second byte, and
for 0x00000001 , 0x03 is inserted after the third byte.
When the decoder decodes, just remove 0x03, which is also called unpacking operation.
Note: In the H264 document, the term EBSP does not appear, but in the official reference software JM of H264, EBSP is used.
EBSP = RBSP + "0x03"
(4) NALU (Network Abstraction Layer Units) network abstraction layer:
each NAL unit is a variable-length byte string with a certain syntax element, including a byte of header information (used to indicate data type),
and several integers The raw byte sequence load (RBSP) of the byte.
A NAL unit can carry a coded slice, I frame, P frame, B frame, a sequence parameter set, or an image parameter set.
H264 adopts NAL unit to be applicable to various networks, and further improves its anti-error ability.
The setting of the serial number can find out which VLC unit is missing. The
redundant coded image makes it possible to get a rougher image if the basic coded image is lost.
NALU = NAL header(1 byte) + RBSP
2. Related concepts of NAL:
The hexadecimal data at the beginning of a piece of H264 stream:
00 00 00 01 67 42 00 1f 9d a8 14 01 6e 9b 80 80
80 81 00 00 00 01 68 ce 3c 80 00 00 00 01 06 e5
01 d3 80 00 00 00 01 65 b8 00 00 03 03 27 5c 27
ff ff c3 d1 40 00 40 7b c9 c9 c9 c9 c9 c9 c9 c9
c9 c9 c9 c9 c9 c9 c9 c9 c9 c9 c9 c9 c9 c9 c9 c9
c9 c9 c9 c9 c9 c9 c9 c9 c9 c9 c9 c9 c9 c9 c9 c9
1. How to judge the frame type (Is it I frame or SPS or PPS): The
official document is as follows:
表7-1－NAL单元类型码
nal_unit_type NAL单元和RBSP语法结构的内容
0 未指定
1 一个非 IDR 图像的编码条带
2 编码条带数据分割块 A
3 编码条带数据分割块 B
4 编码条带数据分割块 C
5 IDR 图像的编码条带
6 辅助增强信息 (SEI)
7 序列参数集
8 图像参数集
9 访问单元分隔符
10 序列结尾
11 流结尾
12 填充数据
13 序列参数集扩展
14..18 保留
19 未分割的辅助编码图像的编码条带
20..23 保留
24..31 未指定
Let's look at the code stream data. The next byte divided by the separator "00 00 00 01" is the NAL unit type code, among which:
bit7: prohibited bit, value 1 indicates syntax error;
bit5-6: reference level;
bit0-4: is the unit type of NAL: the
code stream separator above is followed by 0x67,
0x68, 0x65 0x67-0b0110 0111, bit0-4 converted to decimal is 7, corresponding to the sequence parameter set
0x68-0b0110 1000, bit0 -4 converted to decimal is 8, corresponding to image parameter set
0x65-0b0110 0101, bit0-4 converted to decimal is 5, corresponding to IDR image encoding strip
所以判断NAL单元是否为I帧的方式可以为：(nal_unit_type & 0x1f) == 5
2. Sequence:
Generally speaking, the first frame of data compiled by the encoder is PPS, then SPS, then I frame, P frame, B frame, etc.;
a sequence == one SPS + 1 PPS + one I frame + Several P frames + Several B frames
3. Separator: Separator
is a mark used to distinguish NAL units in a piece of H264 code stream data, namely 0x000001 or 0x00000001.
When such a separator is encountered, the next data is the data of a NAL unit.
4.SPS (Sequence Paramater Set) sequence parameter set:
the SPS stored in the coding sequence for a group of video (Codec Video Sequence) global parameter,
the sequence encoding each frame of data depends on the picture parameter set parameter storage,
general Under the circumstances, the NAL unit of SPS and PPS is usually located at the beginning of the entire code stream. However, in some special cases,
these two structures may also appear in the middle of the code stream. The main reasons may be:
*The decoder needs to be in Start decoding in the middle of the code stream;
*The encoder changes the parameters of the code stream (such as the resolution of the image) during the encoding process;
5. PPS (Picture Paramater Set) image parameter set:
PPS is similar to SPS. It is stored separately in a NAL unit in the H264 code stream,
except that the nal_unit_type value of PPS NAL Unit is 8,
and in the encapsulation format, PPS is usually Together with SPS, it is saved in the file header of the video file.
6.
SEI auxiliary enhancement information SEI is an important technology in the H264 standard, which mainly plays a role of supplement and enhancement.
SEI does not have image data information, but is only a supplement to image data information or video streams.
Some content may be helpful for decoding.
Figure 1. Sequence in the code stream
3. Reference:
https://www.cnblogs.com/wainiwann/p/7477794.html