Detailed Explanation of Video Stream TS Packing Method

Article directory

Detailed Explanation of Video Stream TS Packing Method

The detailed explanation of PS stream packaging was introduced earlier, from which we can see the detailed packaging method of PES. TS stream is actually packaged on top of PES, plus PAT, PMT and other PSI information tables. If you want to know the detailed packaging method of PES, pay attention to Official account: One Zero Warehouse , send: ps to get related articles.
HLS slice files are generally audio and video files encapsulated by TS streams. Here we mainly introduce the encapsulation methods of TS streams.

TS stream header encapsulation

TS stream (Transport Stream) is a further encapsulation of PES packets, the basic unit is TS packets, and the fixed packet size is 188 bytes (or 204 bytes, plus 16 bytes of CRC check data after 188 bytes) , consisting of TS header and payload; its composition is as shown in the figure below:

TS header consists of a 4bytes fixed header followed by an additional field (adaption field), which is explained in byte order as follows:

sync_byte: synchronization code, its size is fixed 8 bits, the value is 0x47
transport_error_indicator: error flag bit, occupying 1 bit, setting it to 1 means that there is at least one uncorrectable error in this packet, generally 0
payload_unit_start_indicator: Payload start flag, used to indicate that the payload of the TS packet contains PES packets or PSI data, occupying 1 bit; if this value is 1, and the payload is PSI data, after the TS header, the payload The start byte will have an adjustment byte point_field; when the TS packet load is PES packet data, the payload_unit_start_indicator is set to 1, indicating that the effective payload of the TS packet starts with the first byte of the PES packet, that is, the TS packet It is the start packet of the PES packet, and there is only one start field of the PES packet in this TS packet; it is set to 0, indicating that the TS packet is not the start packet of the PES packet, but the following data packet; when the TS packet load is For PSI data, the payload_unit_start_indicator is set to 1, indicating that the TS packet contains the first byte of the PSI data segment, that is, the TS packet is the start packet of the PSI Section, and the first byte of the payload of the TS packet With pointer_field; set to 0, indicating that the TS packet does not contain the first byte of the PSI Section, that is, the TS packet is not the start packet of PSI, that is, there is no point_field in the payload, and the beginning of the payload is the data of PSI content. The definition of point_field will be introduced in the following PSI; for empty packets, the payload_unit_start_indicator is set to 0; for example: if the TS packet payload is PAT, when the received TS packet has a payload_unit_start_indicator of 1, it indicates that the TS packet contains PAT header information, parse section_length and continuity_counter from this packet, and then continue to collect TS packets with payload_unit_start_indicator=0, and judge the continuity of continuity_counter, and continuously read the payload in the TS packet (that is, PAT data), use section_length as the end condition for collecting TS packets
transport_priority: Sending priority, set to 1 means that this packet has a higher priority than other packets with the same PID, set to 0, occupying 1 bit;
PID: indicates the data type in the payload, occupying 13 bits; 0x0000 represents PAT, 0x0001 represents CAT, 0x0002-0x000F is reserved, and 0x1FFF represents an empty packet;
transport_scrambling_control: Payload encryption mode flag, occupying 2 bits, 00 means unencrypted;
adaptation_field_control: adjustment field flag, indicating whether the TS header follows the adjustment field or payload data, occupying 2 bits, of which 00 is reserved, 01 means no adjustment field, only payload data, 10 means only adjustment field, no payload, 11 means There is an adjustment field, followed by a payload; for an empty packet, this field should be 01; if adaptation_field_control == 1x, it means the adaptation field field follows; if adaptation_field_control == x1, it means the data_bytes field follows;
Continuity_counter: Continuity count, which increases with each TS packet with the same PID, and returns to 0 after reaching the maximum value; occupying 4 bits, if the value of adaptation_field_control is 00 or 01, this value should not increase; if the flag bit discontinuity_indicator of the adjustment field If the value is 1, this value is not continuous;
Adaptation field: adjustment field, the following fields will exist only when adaptation_field_control == 1x, and the content is as follows:
adaptation_field_length: adjustment field length mark, indicating the length of the adjustment field after this byte, occupying 8 bits; when the value is 0, Indicates that an adjustment byte is inserted in the TS packet, followed by no adjustment field, followed by the payload; when adaptation_field_control == '11', this value is between 0 and 182, when adaptation_field_control == '10', this The value is 183. If the field is not so long, fill in the 0xFF field; the following fields will only appear when adaptation_field_length>0, and the order is as follows:
discontinuity_indicator: discontinuity status indicator, occupying 1 bit, when it is set to 1, it means that the discontinuity status of this TS packet is true;
random_access_indicator: random access indicator, occupying 1 bit;
elementary_stream_priority_indicator: primitive stream data priority indicator, occupying 1 bit, setting 1 means that this primitive stream data has a higher priority than other primitive streams in TS packets with the same PID;
PCR_flag: PCR flag, occupying 1 bit, setting 1 means that the adjustment field contains the PCR field, and setting 0 means there is no PCR field;
OPCR_flag: OPCR flag, occupying 1 bit, setting 1 means that the adjustment field contains the OPCR field, and setting 0 means there is no OPCR field;
splicing_point_flag: splice_countdown flag, occupying 1 bit, setting 1 means that the splice_countdown field is included in the adjustment field, and setting 0 means there is no splice_countdown field;
transport_private_data_flag: transport_private_data flag, occupying 1 bit, when it is set to 1, it means that the adjustment field contains 1 or more private data bytes, and if it is set to 0, there is no such byte;
adaptation_field_extension_flag: adjustment field extension flag bit, occupying 1 bit, setting 1 means that the adjustment field extension field is included, and setting 0 means that there is no extension field; the above
8 bits are identifiers, followed by fields determined according to the value of the identifier, The sequence is as follows:
PRC field: This field only exists when PCR_flag == 1, occupying 48 bits, and the order is as follows:
program_clock_reference_base field: occupies 33 bits;
reserved field: occupies 6 bits;
program_clock_reference_extension field: occupies 9 bits;
OPRC field: This field only exists when OPCR_flag == 1, occupying 48 bits, and the order is as follows:
original_program_clock_reference_base field: occupies 33 bits;
reserved field: occupies 6 bits;
original_program_clock_reference_extension field: occupies 9 bits;
splice_countdown field: this field exists when splicing_point_flag == 1, occupying 8 bits;
transport_private_data field: private data field, this field exists when transport_private_data_flag == 1, occupies N*8bit, byte order is:
transport_private_data_length: Indicates the byte length of private data, occupying 8 bits;
private_data_byte: private data, the length is determined by the previous length field;
adaptation_field_extension field: adjustment field extension field, the occupied length is uncertain, this field exists when adaptation_field_extension_flag == 1, and there are also 3 flag bits in the field to determine whether some fields exist or not. The specific byte order is as follows:
adaptation_field_extension_length: Adjust the length of the field extension field, occupying 8 bits;
ltw_flag: ltw field flag bit, when set to 1, it means that this field exists, occupying 1 bit;
piecewise_rate_flag: piecewise_rate field flag bit, this field exists when the bit is set to 1, occupying 1 bit;
seamless_splice_flag: seamless_splice flag, this field exists when it is set to 1, occupying 1 bit;
Reserved: Reserved field, occupying 5 bits;
Ltw field: This field exists when ltw_flag == 1, occupying 16 bits, which consists of the following two fields
ltw_valid_flag: occupies 1 bit, when ltw_valid_flag == 1, ltw_offset is valid;
ltw_offset: occupies 15 bits;
piecewise_rate field: this field exists when piecewise_rate_flag == 1, occupying 24 bits, and its byte order is as follows:
reserved field: reserved field, occupying 2 bits;
piecewise_rate field: occupies 22 bits; this field is only defined when ltw_flag == 1 and ltw_valid_flag == 1, and when defined, this field is a positive integer;
seamless_splice field: this field exists when seamless_splice_flag == 1, occupying 40 bits; the byte order is as follows:
splice_type field: occupies 4 bits; identifies delay and rate values;
DTS_next_AU[32...30]: occupies 3 bits;
marker_bit field: occupies 1 bit;
DTS_next_AU[29...15] field: occupies 15 bits;
marker_bit: occupies 1 bit;
DTS_next_AU[14...0]: occupies 15 bits;
marker_bit: occupies 1 bit;
stuffing_byte: filling field, fixed to 0xFF;
Payload_bytes: payload field, bytes from PES packets, PSI parts, etc.;

PSI (Program Specific Information Sheet)

After the TS header, it is the content of the payload, which can be the data of the PES packet or the PSI information. The PSI information mainly consists of PAT, PMT, CAT, etc. Here we mainly introduce the two information tables of PAT and PMT; from the above The information described shows that the type of payload is determined by PID. Generally, if PID=0x0000, the payload is PAT; if PID=0x0001, the payload is CAT, and the PID of PMT is specified in PAT; PSI is also
possible There is a special field, the Point_field field: following the header, occupying 8 bits, belonging to the payload, indicating the number of bytes between the beginning of this field and the first byte of the PSI Section in the payload; when payload_unit_start_indicator == 1 , this field only exists; if point_field == 0x00, it means that this byte is followed by the start byte of the PSI Section; this field is included in the payload, and is included in the length of the payload;

PAT: Program Association Table

PAT mainly includes the program number and the PID number of the PMT corresponding to each program, and provides the correspondence between the program number and the PID of the TS packet (PMT packet) containing the program definition information (generally, there is only one in our TS stream) program (channel), so there is generally only one PMT in PAT);
the entire table is composed of many fields; this table may be divided into multiple subsections for transmission, that is, PAT may be divided into multiple TS packets for transmission; The data segment section of PAT consists of the following fields, in order:

table_id field: Indicates the content of the PSI segment, occupying 8 bits. When table_id == 0x00, it means that this segment is a PAT segment. When table_id == 0x01, it means that this segment is a CAT segment. When table_id == 0x02, Indicates that this segment is a PMT segment; in PAT, the value of id is 0x00;
section_syntax_indicator field: Occupies 1 bit, fixed to '1';
'0' field: occupies 1 bit;
Reserved field: reserved field, occupying 2 bits;
section_length field: segment length, occupying 12 bits, of which the first 2 bits are fixed as '00', and the last 10 bits indicate the length of the following section field, including the length of CRC;
transport_stream_id field: TS stream identification id, used to identify this TS stream from other multiplexing in the network, its value is defined by the user, occupying 16 bits;
Reserved: Reserved field, occupying 2 bits;
version_number field: the version number of the entire PAT, occupying 5 bits; when the PAT changes, its value is cyclically accumulated from 0 to 31, when current_next_indicator == 1, version_number is the version number of the current PAT, when current_next_indicator == 0, its The value is the version number of the next PAT;
current_next_indicator field: indicator, occupying 1 bit, when it is set to 1, it means that the current PAT is valid, when it is set to 0, it means that the current PAT is invalid, and the next PAT is valid;
section_number field: the serial number of this section, occupying 8 bits, the serial number of the first section of the PAT should be 0x00, and will be accumulated by 1 with each section in the PAT;
last_section_number field: the serial number of the last section of PAT, that is, the highest serial number value, occupying 8 bits;
Loop：
program_number field: occupying 16 bits, indicating the number of the program available for PMT; if program_number == 0x0000, the next reference PID is network PID, and the value in other cases is defined by the user; in a version_number of PAT, this value cannot be taken A single value is more than once (that is, there can only be a correspondence between a program number and its PMT PID in a PAT section Section);
Reserved: Reserved field, occupying 3 bits;
network_PID field: network PID, occupying 13 bits, this field exists only when program_number == 0x0000; indicates the PID value of the TS packet containing the network information table NIT;
program_map_PID field: occupies 13 bits, when program_number! = 0x0000, this field exists, indicating the PID value of the TS packet of the PMT segment available for the program specified by the program_number; one - program_number should not have multiple program_map_PID assignments, and the value of this program_map_PID is defined by the user, but it cannot be taken as Values reserved for other purposes;
Loop end；
CRC_32 field: CRC check value, occupying 32 bits;

PMT: Program Mapping Table

PMT provides the mapping relationship between the program numbers and the original streams of the programs that make up them. If there are multiple programs in a TS stream, there will be multiple PMT tables with different PIDs. In each PMT, the programs are included. The PID corresponding to the TS packet of different stream types in the original stream; that is, in the PMT, the PID value corresponding to the TS packet that identifies the video stream, audio stream and other data related to this program in the current program; the same as
PAT , PMT may also be divided into one or more Section segments for transmission. PMT consists of many fields, and the order of the fields is as follows:

table_id field: indicates the content of the PSI segment, occupies 8 bits, and is fixed at 0x02 in the PMT;
section_syntax_indicator field: Occupies 1 bit, fixed to '1';
'0' field: occupies 1 bit;
Reserved field: reserved field, occupying 2 bits;
section_length field: segment length, occupying 12 bits, of which the first 2 bits are fixed as '00', and the last 10 bits indicate the length of the following section field, including the length of CRC;
program_number field: program number, occupying 16 bits, which specifies the program number corresponding to this PMT, and only one program definition can be included in the PMT section of a TS;
Reserved field: reserved field, occupying 2 bits;
version_number field: the version number of this PMT segment, occupying 5 bits, it will accumulate 1 with the change of this segment information, and return to 0 cycle after reaching 31; the version number corresponds to the definition of a single program, that is, corresponds to a single Segmentation; when current_next_indicator == 1, the value of number is the version number of the current PMT segment; when current_next_indicator == 0, the value of number is the version number of the next available PMT segment;
current_next_indicator field: indicator, occupying 1 bit, when it is set to 1, it means that the current PMT segment is valid, when it is set to 0, it means that the current PMT segment is invalid, and the next PMT segment is valid;
section_number field: occupies 8 bits and is fixed at 0x00;
last_section_number field: occupies 8 bits and is fixed at 0x00;
Reserved field: reserved field, occupying 3 bits;
PCR_PID field: the PID value of the TS packet where the PCR is located, occupying 13 bits; it indicates the PID value of the TS packet containing the PCR field in the program specified by program_number; if the program definition of a private stream has no PCR related to it, this field should be set 0x1FFF;
Reserved field: reserved field, occupying 4 bits;
program_info_length field: length field, occupying 12 bits; the first 2 bits are fixed to 00, and the last 10 bits indicate the number of bytes of the descriptor after this field;
Descriptor field: program description information, the length is determined by the previous field;
LOOP：
stream_type field: stream type field, occupying 8 bits; specifies the type of program stream in the payload of the TS packet specified by elementary_PID, that is, the audio and video coding type; commonly used stream_type enumeration: 0x10-MPEG-4 video stream; 0x1B-H264;0x24-H265;0x80-SVAC;0x90-G711;0x92-G722.1;0x99-G.729;0x9B-SVAC;0x0f-AAC
Reserved field: reserved field, occupying 3 bits;
elementary_PID field: PID field, occupying 13 bits; indicates the PID value of the TS packet carrying the relevant original stream ES, that is, the PID value of the TS packet such as video packet and audio packet;
Reserved field: reserved field, occupying 4 bits;
ES_info_length field: length field, occupying 12 bits; the first 2 bits are fixed at 00, and the last 10 bits represent the description field length of the original stream ES of related programs after this field;
Descriptor field: ES stream description information, the length is determined by the previous field;
LOOP End；
CRC_32 field: CRC check value, occupying 32 bits;

Analysis of TS packet capture instance

Through the ts file extracted from wireshark, the analysis is performed through Elecard StreamEye Tools software (soft solutions can follow the official account: Yilingcang , send video stream analysis , and obtain tools) as follows: As

can be seen from the above figure, the process of TS analysis:

According to PID=0, find and analyze the PAT, and analyze the number of PMTs and the PID corresponding to the PMT according to the PMT associated with the PAT program. Generally, there is only one PMT by default.
According to the PID corresponding to the PMT, find and parse all the streams, stream types, and stream PIDs contained in the program mapping table. Here, we know the stream type and PID for each stream
According to the meaning of stream type, we find the PID corresponding to the stream type we need to analyze, and find the real audio and video stream according to this PID

The decapsulation and packaging of TS can be designed according to this process.

Summarize

There are many definitions of TS. In fact, when we actually develop codes for TS encapsulation and decapsulation, we need to pay attention to very few fields. Many fields of encapsulation and decapsulation are default values, and most of the optional fields are not available. Refer to ffmpeg related code.