Table of contents
Four. WebRTC ULPFEC implementation source code analysis
2. The calling process for generating FEC packets
3. Rules for the number of redundant packets
4. Protection Assignment Rules
5. XOR generates FEC packet payload
6. Pack and send in RED format
7. Receive FEC packets and recover lost media packets
I. Introduction
UDP is usually used to transmit real-time audio and video data in the network. Due to the problem of packet loss during transmission, the heard sound may be intermittent or the video may freeze. For weak network scenarios, appropriate redundant packets can be added to fight against it. Caton problems caused by packet loss.
The basic principle of FEC is to generate certain redundant data through the original data. If the original data is lost during transmission, the receiving end can also restore the original data through the redundant data.
This article introduces the basic theory and message format of ULPFEC, and introduces the source code implementation related to WebRTC ULPFEC.
2. Basic principle of FEC
The basic principle of FEC is to generate redundant data through original data according to certain operations. Redundant data can be generated through XOR, or redundant data can be generated through Reed-Solomon algorithm. Taking ULPFEC as an example, it uses XOR operation to generate For redundant data, the characteristic of XOR is that the difference is 1, and the same is 0.
As shown in the figure below, Data1 and Data2 are bitwise XORed to generate redundant data R. Since 0 ^ 0 = 0, 1 ^ 1 = 0, 0 ^ 1 = 1, 1 ^ 0 = 1, the value of R is as follows Show.
Originally, the sending end transmits Data1, Data2 to the receiving end. If Data1 or Data2 is lost during the transmission, the receiving end cannot collect the complete data and can only wait for retransmission. Now the sending end sends Data1, Data2, R to the receiving end at the same time. At the end, assuming that Data1 or Data2 is lost, the lost Data can be restored by XORing R with another unlost Data without waiting for the sender to retransmit.
Here is just an example of Data1 and Data2. The same is true for multiple data packets. In actual scenarios, how many data packets are a group and the number of redundant packets generated is usually determined by combining packet loss rate, code rate, frame rate and other information. .
3. ULPFEC
FEC includes RedFec, ULPFec, and FlexFec, etc. WebRTC uses ULPFec and uses the Red encapsulation format to encapsulate redundant packets.
1. ULPFEC basic theory
ULPFEC (Uneven Level Protection, Unequal Protection) can provide different levels of protection for different data packets, as described below as an example.
The original data packets are Packet A, B, C, D, the redundant packets are ULPFEC Packet #1 and #2, and the redundant packets provide protection for the original data packets.
ULPFEC Packet #1 provides level 0 protection for the L0 length of Packet A and B (note that it does not protect the entire package of Packet A and B), ULPFEC Packet #2 has two protection levels, and level 0 protection is for Packet C and D L0 length provides protection, and level 1 protection provides protection for Packets A, B, C, and D.
Assuming that Packet A is lost during the transmission of Packet A, B and ULPFEC Packet #1, the data of the first L0 length of Packet A can be restored according to ULPFEC Packet #1 and Packet B, and the data after the length of Packet A L0 cannot be recovered ( ULPFEC Packet #2 is not considered here for now), because ULPFEC Packet #1 only provides level 0 protection for Packet A and B, and L0 < min(len(Packet A), len(Packet B)), if you think Incomplete data packets are useless for applications, you can also adjust the length of L0 to max(len(Packet A), len(Packet B)), this is usually a trade-off between bandwidth and protection, the more redundant Larger means stronger anti-packet loss capability, but also consumes more bandwidth.
The level 0 protection for Packet C, D and ULPFEC Packet #2 is the same as for Packet A, B and ULP FEC Packet #1, the difference is that for ULP FEC Packet #2, it is Packet A, B, C, D at the same time Provides level 1 protection (that is, the L1 length part), assuming that any packet of A, B, C, D is lost during transmission, and ULPFEC Packet #2 is not lost, then the L1 length data part of the lost packet can also be recovered.
For L0 protection, two original data packets correspond to one redundant packet, and for L1 protection, four original data packets correspond to one redundant packet. Obviously, L0 has higher redundancy and stronger protection.
2. ULPFEC message format
The format of the FEC message is as shown above. First is the RTP Header, followed by a fixed 10-byte FEC Header, and then one or more layers of FEC Level Header and Payload.
RTP Header
The RTP Header of ULPFEC is only used when the FEC message is sent through a separate data stream. Its format and field meaning still follow the definition of RFC3550 . If you are not familiar with the meaning of the RTP Header field, please read this article . The following is only for Some fields are further explained.
M: Mark bit, should be set to 0 in ULPFEC RTP Header
PT: The payload type of the FEC packet, which can be dynamically selected and determined
SSRC: Same as the SSRC of the protected media stream
SN (sequence number): It must be 1 greater than the sequence number of the previously transmitted FEC message
TS (timestamp): The timestamp must be set to the media RTP timestamp when the FEC message is transmitted
FEC Header
E: Reserved extension bit, currently set to 0
L: Indicates whether to use a long mask. If L is set to 0, it means that the mask of FEC Level Header is 2 bytes. If L is set to 1, it means that the mask of FEC Level Header is 6 bytes.
P / X / CC / M / PT recovery: through the exclusive OR operation of the P / X / CC / M / PT bits of the RTP headers of all media packets protected by FEC packets
SN base: the smallest sequence number among all RTP media packets protected by this FEC packet
TS recovery: The timestamp of all RTP media packets protected by this FEC packet is obtained after XOR operation
length recovery: This value is obtained after the payload length of all media packets protected by this FEC packet is subjected to XOR operation
FEC Level Header
Protection Length: the protection length corresponding to the protection level
mask: mask, if the i-th bit in the mask is set to 1, it means that the media packet with SN base + i sequence number is associated with this FEC packet, note that the position of the most significant bit of the mask corresponds to i = 0, which is the 16th bit in the above figure The position of the least significant bit corresponds to i = 15 or i = 47 (depending on whether L in the FEC Header is 0 or 1)
We use two examples to illustrate how the various fields of the FEC message are determined, assuming that the FEC stream is sent through a separate RTP session, and we are going to send four media packets A, B, C, D (SSRC=2) , their serial numbers are 8, 9, 10, 11 respectively, and their timestamps are 3, 5, 7, 9 respectively, the Payload Type of Packet A and C=11, the Payload Type of Packet B and D=18, and the payload of A Length is 200, B is 140, C is 100, and D is 340.
Example 1: Use a single protection level to protect the full length of four Packets (that is, the protection of the entire packet), and any loss of Packet A/B/C/D can be recovered through FEC Packet.
See Figure 7 below for FEC RTP Header, Figure 8 below for FEC Header, and Figure 9 below for FEC Level Header.
Version is fixed at 2, Padding is 0 means no padding, Extension is 0 means there is no RTP extension header, Marker is fixed at 0, PT is 127 (we assume Payload Type=127 to represent FEC packets), the starting sequence number is 1, TS is the media RTP timestamp when the FEC packet is sent (D timestamp is 9, so TS = 9), SSRC is the same as the SSRC of the protected media stream, so it is 2.
E is reserved extension bit, the value is fixed at 0, L is 0 means use short length mask, the value of P / X / CC / M / PT is based on the P / X / CC / M / PT XOR, SN base is the minimum serial number min(8, 9, 10, 11) of all protected RTP packets, TS recovery value is obtained by XOR operation based on the time stamps of all protected RTP packets, length recovery is obtained after XOR operation of the payload lengths of all media packets to be protected.
The FEC Level Header part is shown above, Protection Length is the protection length of the corresponding level, L0 provides protection of the maximum length of 4 Packets, so the value is 340 (the payload length of Packet D), and the mask value is 0b1111000000000000, because the FEC associated The sequence numbers of the media packets are 8, 9, 10, 11, and SN base = 8.
Example 2: Example with two protection levels, L0=70, L1=90
Two FEC packets ULP #1 and #2 are generated as above, ULP #1 protects the L0 length of Packet A and B, ULP #2 protects the L0 length of Packet C and D, and protects the L1 of Packet A, B, C, and D at the same time length.
The RTP Header of ULP #1 is shown in Figure 11 below, the FEC Header is shown in Figure 12 below, and the FEC Level Header is shown in Figure 13 below.
The RTP Header of ULP #2 is shown in Figure 14 below, the FEC Header is shown in Figure 15 below, the FEC Level Header (L0) is shown in Figure 16 below, and the FEC Level Header (L1) is shown in Figure 17 below.
FEC Level Payload
FEC Level Payload is generated by bit-wise XOR of data corresponding to the length of the protected package Level. If the length of some packages is not enough for the coverage of the corresponding protection level, it will be filled with 0 at the end.
Four. WebRTC ULPFEC implementation source code analysis
1. Create FEC object
The ULPFEC function is enabled by default for WebRTC video (h264 is not enabled). The following is the creation process of FEC-related objects. FEC has two important classes, one is FecController, which is the controller of FEC, and is used to calculate parameters such as FEC protection factors. VideoFecGenerator, which is the operation object of FEC, is used to generate FEC redundancy for media packets. It uses FecController. WebRTC also includes the implementation of FlexFec, but ULPFEC is used by default.
2. The calling process for generating FEC packets
After the encoder finishes encoding the frame image, it will call back the RtpVideoSender::OnEncodedImage function, then call RTPSenderVideo::SendEncodedImage, and then call RTPSenderVideo::SendVideo. In the SendVideo function, if the VideoFecGenerator object is not empty, it will call AddPacketAndGenerateFec, and then perform EncodeFec action.
3. Rules for the number of redundant packets
In the EncodeFec function, NumFecPackets will be called to obtain the number of redundant packets that need to be generated, which is determined according to the current number of media packets to be protected and the protection factor.
protection_factor is passed in when the EncodeFec function is called, and its value is CurrentParams().fec_rate. CurrentParams will select keyframe_params or delta_params according to whether the currently processed frame is a keyframe, and current_params_ is assigned by *pending_params_.
The value of pending_params_ is updated in UlpfecGenerator::SetProtectionParameters, and the call stack of SetProtectionParameters is traced back. The call flow is: RtpVideoSender::OnBitrateUpdated -> FecControllerDefault::UpdateFecRates -> RtpVideoSender::ProtectionRequest -> UlpfecGenerator::SetProtectionParameters.
FecControllerDefault::UpdateFecRates will calculate the required protection factor based on the code rate, packet loss rate and other information. For the specific logic, see VCMFecMethod::ProtectionFactor.
Before introducing the calculation logic of the FEC protection factor, we first introduce kFecRateTable, which is a one-dimensional array table used to determine the protection factor, with a length of 6450. It is usually regarded as a 50*129 two-dimensional array table. If the effective code rate of the frame and the current packet loss rate are known, a subscript position can be determined to obtain the corresponding protection factor. The row is obtained by calculating the effective bit rate of the frame, the span between rows is 5kbps, the column is determined by the packet loss rate, and the 128th column indicates a 50% packet loss rate.
The following is the code logic for calculating the protection factor. If packetLoss == 0 means no packet loss, and protectionFactor is 0, no redundant packets are required. If the packet loss rate is greater than 0, calculate the effective bit rate of the frame and combine the packet loss rate. Get the protection factor from kFecRateTable.
4. Protection Assignment Rules
Assuming that M media packets need to be protected by N FEC packets, then how to determine the protection distribution rules, which FEC packets should protect which media packets are the content to be discussed in this section.
There are two mask tables in WebRTC, kPacketMaskRandomTbl and kPacketMaskBurstyTbl. These two tables are used for the protection allocation rules when M media packets need N FEC packet protection in the case of random packet loss and burst packet loss respectively. WebRTC uses by default It is a model of random packet loss. It does not check whether the current packet loss is random or burst packet loss for dynamic selection.
The random packet loss mask table is shown below, which defines the mask rules when 1-12 media packets are protected.
kPacketMaskRandomM contains kMaskRandomM_1...kMaskRandomM_N items (N <= M). Let's take 1 FEC packet to protect 1 media packet as an example. The mask is 0b1000000000000000 (0x80, 0x00), that is, the FEC packet protects SN+0 The media packet with the sequence number, if one FEC packet protects two media packets, its mask is 0b1100000000000000 (0xc0, 0x00), that is, the FEC packet protects the media packets with SN+0 and SN+1 sequence numbers, if there are two The FEC packet protects 3 media packets, the mask of the first FEC packet is 0b1100000000000000, the mask of the second FEC packet is 0b10100000000000000, that is, the first FEC packet protects the media packets with SN+0 and SN+1 serial numbers, The second FEC packet protects the media packets with sequence numbers SN+0 and SN+2. For other combinations, refer to the mask table.
5. XOR generates FEC packet payload
After determining the number of redundant packets and the protection allocation rules, it is necessary to generate redundant packets according to the XOR of media packets. The code flow is as follows. GenerateFecPayloads is used to generate FEC packets, but the FEC in this step is not the final state, and FinalizeFecHeaders needs to be called to adjust the FEC some info on the header.
The GenerateFecPayloads process is analyzed as follows. First, determine whether the current media packet is protected by this FEC packet. If not, skip to the next media packet processing. If it needs to be protected, if this FEC packet has not been filled with media packet data, then do it directly. For some copying (see the following code for how to copy from the media package), if the FEC package has already filled in the content of the media package, use the media package data and the existing data in the FEC package to do XOR operation.
Then adjust some field values of the FEC Header. The rules are consistent with those described in the RFC document and will not be described here.
6. Pack and send in RED format
RED encoding is a redundant encoding method, which allows data of different encodings to be inserted into the same RTP packet. The format is as follows, see RFC2198 for details .
After WebRTC enables red and ulpfec, it will insert the media packet data into the RED RTP packet for the media packet, the PT of the RTP Header is the Payload Type used by RED, and the block PT is set as the Payload Type used by the media packet. For the FEC packet, it puts The FEC packet data is stuffed into the RED RTP packet, the PT of the RTP Header is the Payload Type used by RED, and the block PT is set to the Payload Type used by ulpfec. Note that the media package and the FEC package in WebRTC are packaged separately, not in the same RED package. The figure below shows that RED is stuffed with two PT formats at the same time.
7. Receive FEC packets and recover lost media packets
Using FEC packets to restore lost media packets is the reverse operation of using media packets to generate FEC packets.
First, after receiving the RTP packet, it will enter RtpVideoStreamReceiver::ReceivePacket, and if it is judged as the payloadType of the red packet, it will enter ParseAndHandleEncapsulatingHeader for processing.
ParseAndHandleEncapsulatingHeader first executes AddReceivedRedPacket, which stuffs the received media packet and FEC into received_packets_, and marks the corresponding distinction, and then executes the ProcessReceivedFec logic. On the one hand, it calls back the media packet to RtpVideoStreamReceiver::OnRecoveredPacket, which is equivalent to performing normal reception The processing flow of non-RED format media packets, and then execute DecodeFec to try to recover the lost media packets according to the FEC packets.
The logic of DecodeFec is as follows, first execute InsertPacket to insert the media packet and FEC packet into the appropriate list, and then execute AttemptRecovery to try to recover the lost data packet.
For media packets, it is inserted into the recovered_packets list. For FEC packets, it will analyze the media packets protected by the FEC packet according to the mask list, and then hang the media packets that need to be protected under its own protected_packets list, and then insert the FEC packets into received_fec_packets_ .
The logic of actually trying to recover media packets is in the AttemptRecovery function, it first traverses received_fec_packets_, judges the absence of media packets protected by this FEC packet, and performs the following different processing.
1. If packets_missing is 0, it means that the received media packet is complete, and this FEC packet can be deleted
2. If packets_missing is greater than 1, it means that the condition is immature and you need to continue to wait and skip processing first (because according to the XOR principle, one FEC packet can only recover one media packet lost in the media packet group)
3. If packets_missing is equal to 1, it means that the recovery conditions are met, just call RecoverPacket XOR to recover the media package, and then insert the recovered media package into recovered_packets
For the recovered media packet, call back to RtpVideoStreamReceiver::OnRecoveredPacket, that is, process it as normal when receiving a media packet in a non-RED format.
So far, how WebRTC packs FEC packets according to media packets, how to determine the number of packets, how to determine protection rules, the packaging format when sending, and how to restore media packets according to FEC packets after receiving have been introduced.
V. References
RTC3550（RTP: A Transport Protocol for Real-Time Applications）
RFC2198（RTP Payload for Redundant Audio Data）
RFC5109（RTP Payload Format for Generic Forward Error Correction）