1. Background introduction

There are three redundancy methods implemented by WebRTC: UlpFEC (rfc5109), FlexFEC (rfc8627), and inbandFEC (used by opus audio).

The principles of UlpFEC and FlexFEC are both to XOR a group of M packets to generate N (N is the redundancy of FEC) FEC packets and package them out. If any N pieces of this group of messages are lost, they can be recovered through this group of (MN) messages+FEC redundant packets, increasing the protection range of the FEC redundant messages. For example, the following schematic diagram: D is a media package, R is a redundancy package, and the redundancy shown in the figure is 2.

1. Schematic diagram of packaging at the sending end

2. Schematic diagram of network packet loss

3. Schematic diagram of message recovery

二、FlexFec VS UlpFEC

	Media pack packaging format	FEC SSRC	FEC Sequence
UlpFEC	rfc2198+rfc5109	Same media message SSRC	Share Sequence with media packets
FlexFEC	rfc8627	SSRC alone	Use a separate Sequence

1. WebRTC H264 has only one layer of Sequence in the encapsulation. It relies on the Sequence to continuously judge whether it is a complete video frame. There is no more information encapsulation. As a result, if the FEC message data is in the middle of a frame of packets, such judgment logic will be broken. has a problem. VPX, in addition to Sequence, also has more boundary information and type information such as PictureId, which can decode FEC packets very well. So WebRTC only enables UlpFEC when VPX is used by default.

MaybeCreateFecGenerator
->ShouldDisableRedAndUlpfec
->PayloadTypeSupportsSkippingFecPackets

By default, only VPX supports UlpFEC redundant encoding.

bool PayloadTypeSupportsSkippingFecPackets(const std::string& payload_name,
                                           const WebRtcKeyValueConfig& trials) {
  const VideoCodecType codecType = PayloadStringToCodecType(payload_name);
  if (codecType == kVideoCodecVP8 || codecType == kVideoCodecVP9) {
    return true;
  }
  if (codecType == kVideoCodecGeneric &&
      absl::StartsWith(trials.Lookup("WebRTC-GenericPictureId"), "Enabled")) {
    return true;
  }
  return false;
}

2. In fact, the SSRC of UlpFEC is the same as the SSRC of media packets, and Sequence and media packets are shared. This will make it impossible to distinguish between media packets and FEC packets when UlpFEC is enabled, and make FEC redundant when NACK is enabled. The packet will also be retransmitted with NACK, which will lead to a certain degree of bandwidth waste.

So at present, everyone prefers FlexFEC redundant coding, and UlpFEC redundant coding is less used.

3. Principle of FlexFEC

1. Redundant mode

According to the proposal of rfc8627 protocol, FlexFEC can XOR in 1D row, column, and 2D array, three encoding modes.

1) 1D row XOR mode

2) 1D column XOR mode

3) 2D array XOR mode

But the Webrtc source code only implements MaskRandom, MaskBursty or 1D column XOR redundancy mode.

For details, please refer to PacketMaskTable::LookUp function implementation

2. RTP packet protocol

1) Complete RTP message format

2) FEC Header definition

webrtc does not fully follow the RFC protocol, but customizes a set of protocols by itself. The format is defined in the flexfec_header_reader_writer.h header file.

Mask mask length description

// Size (in bytes) of packet masks, given number of K bits set.
constexpr size_t kFlexfecPacketMaskSizes[] = {2, 6, 14};

The mask is used to identify the list of FEC redundant packet protection packets. The formula for calculating the protected serial number list is:

SN base_i + mask corresponds to the bit position number of 1. Refer to the ForwardErrorCorrection::InsertFecPacket function to calculate the protection queue sequence number of FEC redundant packets:

3. SDP negotiation

From the above example, it can be seen that the pt value of the media video RTP packet is 100, the PT value of the flexfec redundant packet is 110, the ssrc of the media video RTP packet is 1234, and the ssrc of the flexfec redundant packet is 2345.

The implementation code corresponding to webrtc is as follows:

1. PT value determination GetPayloadTypesAndDefaultCodecs

2. Confirmation of ssrc AddFecFrSsrc

4. Code implementation

1. Implementation of FlexFEC encoding

1) FlexFEC generation and push pacer queue call stack

2) FlexFEC message generation process

UlpfecGenerator::AddPacketAndGenerateFec function reading instructions:

Currently webrtc is limited to only support 48-bit masks (because Kbit needs to occupy space, if 48 packets are redundant, the maskhead is 14 bytes), there is a deficiency in the code, that is, if the number of single-frame video packets is greater than 48, Subsequent packets will not be pushed to the media_packets_ queue, and will not participate in redundancy.

WebRTC is designed in this way considering that UlpFEC does not allow FEC messages to be sent in the middle of a frame packet, otherwise it will affect the integrity judgment of the frame by the receiving end. But currently FlexFEC uses separate SSRC and Sequence parameters, this problem can be solved naturally. But the WebRTC source code did not update the design synchronously. As a result, there may be a probability that some packets in the next frame cannot be protected by FEC.

webrtc QOS method 2.3 (introduction to flexfec rfc8627)