Table of contents

1. The receiving end records and feeds back the received packets

2. The sender performs bandwidth estimation and congestion control based on the packet reception feedback

(1) Bandwidth estimation based on delay gradient

(2) Bandwidth estimation based on packet loss rate

I. Introduction

The link bandwidth in network transmission is limited. In order to avoid sending overloaded data to the link and causing network congestion, we need to estimate the bandwidth and make adjustments based on the estimated bandwidth to avoid network congestion.

In "WebRTC GCC Congestion Control Algorithm (REMB-GCC)", we summarized the REMB-GCC congestion control algorithm, and mentioned at the end of the article that Google has launched TFB-GCC to replace REMB-GCC. The principle of TFB-GCC is also based on the delay gradient and packet loss rate to estimate the bandwidth and avoid network congestion. The difference is that the logic is calculated at the sending end, and the receiving end only feeds back the receiving status of the packet (whether received, and difference in the reception time of the previous packet).

This article mainly explains the principles and implementation of TFB-GCC. If you are not clear about the principles of estimated bandwidth based on delay gradient and packet loss, you can read this article first .

2. Principle of TFB-GCC

The above is the architecture diagram of TFB-GCC, the left side is the sending end part, and the right side is the receiving end part.

The receiving end is responsible for recording the arrival of data packets at the sending end, and constructing an RTCP message to feed back to the sending end. It does not perform the logic of delay gradient calculation.

After receiving the RTCP feedback message, the sending end estimates the bandwidth As according to the packet loss rate, and estimates the bandwidth Ar according to the delay gradient. The final estimated bandwidth is the smaller value of the two A=min(As, Ar) , to perform bandwidth estimation and congestion control.

1. The receiving end records and feeds back the received packets

If WebRTC wants to use TFB-GCC, it needs to enable the RTP message extension field (transport-wide sequence number) and use RTCP RTPFB TW message feedback (transmission bandwidth feedback message). For details about RTP and RTCP protocol, please refer to the corresponding link article.

（1）transport-wide sequence nunmber

The structure of the RTP extension field transport-wide sequence number is as follows. It is a one-header extension header with a length of 2 bytes, which can be understood as the channel sequence number.

Question: Why do you need to use transport-wide sequence number?

A channel often transmits audio and video at the same time. When we estimate the bandwidth, we need to estimate the bandwidth of the entire channel, instead of only combining the situation of sending and receiving packets of the audio stream or the situation of sending and receiving packets of the video stream. When the audio stream and video stream are sent, the initial sequence number of the RTP packet is different, and the growth rate of the sequence number is also different. Therefore, we put a channel sequence number on the audio packet and video packet when sending the packet, and count them uniformly. It is also convenient for the receiving end to respond to the reception of the channel packet.

The following is the RTP sending packet information captured by Wireshark. payload-type: 96 is a video stream packet, and payload-type: 111 is an audio stream packet. You can see that the transport-wide sequence number of these four packets is increasing .

Remarks: The above uses ID=3 to represent the transport-wide-cc extension header, a=extmap:3 http://www.ietf.org/id/draft-holmer-rmcat-transport-wide-cc-extensions-01.

(2) RTCP RTPFB TW message

RTCP RTPFB TW transmission bandwidth feedback message format is as follows (PT=205, FMT=15).

base sequence number : record the transport-wide sequence number of the first RTP packet to be fed back

packet status count : The feedback message contains the arrival status of how many RTP packets

reference time : the reference time (24bit) of receiving the first packet of the feedback message from the receiving end, and its value unit is 64ms

fb pkt.count : The number of feedback packets sent, which is equivalent to the sequence number of RTCP RTPFB packets

packet chunk : Record the arrival status of the RTP packet sent by the sender. This structure can be expressed as Run length chunk and Status vector chunk according to the value of bit 0

Run length chunk structure

T (chunk type): This bit is 0, indicating that this is a Run length chunk

S (packet status symbol): marks the arrival status of the packet

00：Packet not received

01：Packet received, small delta

10：Packet received, large or negative delta

11：Reserved

Run Length: Length, indicating how many consecutive packets are in the same arrival state

The following Run length chunk indicates that 221 consecutive packets have not been received.

Status vector chunk structure

T (chunk type): This bit is 1, indicating that this is a Status vector chunk

S (symbol size): If this bit is 0, it means that it only contains two states of packet not received / packet received, so that the 14bit symbol list can represent the status of 14 packets; if this bit is 1, it means that 2 bits are used to represent the status of the packet , so that the symbol list can represent the status of 7 packages

symbol list: identifies the status of a series of packages

Remarks: When the receiving status of many packets is consistent, for example, they are all Received and the arrival interval is SmallDelta, use Run length chunk to represent the receiving status of these packets. When the receiving status of the packets is inconsistent, Run length chunk cannot be used to indicate The same receiving status, the Status vector chunk is used at this time.

In the Status vector chunk shown in the figure above, the S bit is 0, indicating that 1 bit in the symbol list represents the receiving status of a packet, 0 means not received, and 1 means received, so the above figure indicates that 1 packet has not been received, and the next 5 The packet was received, the next 3 packets were not received, the next 3 packets were received, and the last 2 packets were not received.

In the Status vector chunk shown in the above figure, the S bit is 1, indicating that 2 bits in the symbol list represent the receiving status of a packet, so the above figure indicates that the first packet is not received, and the second packet is received (w/ o timestamp), the next 3 packets are received, and the last 2 packets are not received.

recv delta : The arrival time interval of two RTP packets before and after, the unit value represents 250us, if the arrival time interval is <= 63.75ms, it is considered as SmallDelta, and 1 byte is used, and if it is greater than 63.75ms, it is considered as LargeDelta, and 2 bytes are used.

2. The sender performs bandwidth estimation and congestion control based on the packet reception feedback

(1) Bandwidth estimation based on delay gradient

As shown in the figure below, the sender sends two data packets at T(i-1) and T(i), and the receiver receives these two data packets at t(i-1) and t(i) respectively. Time gradient d(t(i)) = [t(i) - t(i-1)] - [T(i) - T(i-1)].

In ideal network transmission, d(t(i)) should be 0. If the network is congested, it will take longer for the packet sent at T(i) to be received by the receiving end. At this time, d(t(i) )) is greater than 0, if d(t(i)) is greater than 0 and getting larger, it means that the network congestion is more serious, if d(t(i)) is greater than 0 but getting smaller, it means that the network congestion is improving .

We use WebRTC code to illustrate how the sender estimates bandwidth and performs congestion control processing logic based on RTCP feedback messages and packet loss rate. For TFB-GCC, bandwidth estimation based on delay gradient mainly includes: Arrival-time filter , TrendlineEstimator.

a. Arrival-time filter

When the RTCP RTPFB TW message is received, the RtpTransportControllerSend::OnTransportFeedback function will be called. This function first calls TransportFeedbackAdapter::ProcessTransportFeedback to obtain the arrival status information of the packet, and then calls GoogCcNetworkController::OnTransportPacketsFeedback to calculate the bandwidth based on the delay gradient based on the arrival status information of the packet. Estimate, and finally update the estimated bandwidth value to Pacer, encoding and other modules, the specific logic is as follows.

TransportFeedbackAdapter::ProcessTransportFeedback mainly calls ProcessTransportFeedbackInner to obtain the relative arrival time of a group of packets according to the content of the RTCP RTPFB TW message, which is used to calculate the delay gradient of the packet.

std::vector<PacketResult>
TransportFeedbackAdapter::ProcessTransportFeedbackInner(
    const rtcp::TransportFeedback& feedback,
    Timestamp feedback_receive_time) {
  // Add timestamp deltas to a local time base selected on first packet arrival.
  // This won't be the true time base, but makes it easier to manually inspect
  // time stamps.
  if (last_timestamp_.IsInfinite()) {
    current_offset_ = feedback_receive_time;
  } else {
    // TODO(srte): We shouldn't need to do rounding here.
    const TimeDelta delta = feedback.GetBaseDelta(last_timestamp_)
                                .RoundDownTo(TimeDelta::Millis(1));
    // Protect against assigning current_offset_ negative value.
    if (delta < Timestamp::Zero() - current_offset_) {
      RTC_LOG(LS_WARNING) << "Unexpected feedback timestamp received.";
      current_offset_ = feedback_receive_time;
    } else {
      current_offset_ += delta;
    }
  }
  last_timestamp_ = feedback.GetBaseTime();

  std::vector<PacketResult> packet_result_vector;
  packet_result_vector.reserve(feedback.GetPacketStatusCount());

  size_t failed_lookups = 0;
  size_t ignored = 0;
  TimeDelta packet_offset = TimeDelta::Zero();
  for (const auto& packet : feedback.GetAllPackets()) {
    int64_t seq_num = seq_num_unwrapper_.Unwrap(packet.sequence_number());

    if (seq_num > last_ack_seq_num_) {
      // Starts at history_.begin() if last_ack_seq_num_ < 0, since any valid
      // sequence number is >= 0.
      for (auto it = history_.upper_bound(last_ack_seq_num_);
           it != history_.upper_bound(seq_num); ++it) {
        in_flight_.RemoveInFlightPacketBytes(it->second);
      }
      last_ack_seq_num_ = seq_num;
    }

    auto it = history_.find(seq_num);
    if (it == history_.end()) {
      ++failed_lookups;
      continue;
    }

    if (it->second.sent.send_time.IsInfinite()) {
      // TODO(srte): Fix the tests that makes this happen and make this a
      // DCHECK.
      RTC_DLOG(LS_ERROR)
          << "Received feedback before packet was indicated as sent";
      continue;
    }

    PacketFeedback packet_feedback = it->second;
    if (packet.received()) {
      packet_offset += packet.delta();
      packet_feedback.receive_time =
          current_offset_ + packet_offset.RoundDownTo(TimeDelta::Millis(1));
      // Note: Lost packets are not removed from history because they might be
      // reported as received by a later feedback.
      history_.erase(it);
    }
    if (packet_feedback.network_route == network_route_) {
      PacketResult result;
      result.sent_packet = packet_feedback.sent;
      result.receive_time = packet_feedback.receive_time;
      packet_result_vector.push_back(result);
    } else {
      ++ignored;
    }
  }

  if (failed_lookups > 0) {
    RTC_LOG(LS_WARNING) << "Failed to lookup send time for " << failed_lookups
                        << " packet" << (failed_lookups > 1 ? "s" : "")
                        << ". Send time history too small?";
  }
  if (ignored > 0) {
    RTC_LOG(LS_INFO) << "Ignoring " << ignored
                     << " packets because they were sent on a different route.";
  }

  return packet_result_vector;
}

After obtaining the relative arrival time of the packet, call DelayBasedBwe::IncomingPacketFeedbackVector to analyze the delay gradient change. The key calling process is: DelayBasedBwe::IncomingPacketFeedbackVector -> DelayBasedBwe::IncomingPacketFeedback -> InterArrival::ComputeDeltas.

b. Trendline estimator

In REMB-GCC, the Kalman filter is used to calculate the change of the delay gradient, while in TFB-GCC, the linear filter is used to calculate the change trend of the cumulative delay gradient, that is, a bunch of sample points are fitted by the least square method ( x, y), the trend of change can be judged by the slope of the line. Substituting into the fitting line equation, x is equivalent to time, and y is equivalent to the smoothed cumulative delay gradient.

After obtaining the slope of the linear filter, call TrendlineEstimator::Detect to judge the current bandwidth usage status. TrendlineEstimator::Detect judges whether the current bandwidth is in the overuse/normal/underuse state according to the relationship between the trend trend of the cumulative delay gradient and the dynamic threshold value, and the dynamic threshold value threshold_ will be explained in the next subsection Adaptive threshold.

a. If trend > threshold_, it means that the network congestion queue is increasing and is currently in a congested state. If the congestion duration is greater than overusing_time_threshold_, and the delay gradient is greater than the previous delay gradient, it is judged to be in an overuse state. Note that it is not once greater than the threshold It is judged to be in overuse, it needs to last for a period of time and the delay gradient is getting larger before it is judged to be in overuse

b. If trend < -threshold_, it means that the network congestion queue is getting smaller, the congestion situation is improving, and it is judged to be in the underuse state

c. If -threshold_ <= trend <= threshold_, the judgment is in normal state

（3）Adaptive threshold

As mentioned above, TrendlineEstimator judges the current bandwidth usage status by comparing the change of the cumulative delay gradient with the threshold value. In an ideal network, the delay gradient is 0, but under normal bandwidth occupancy, the delay gradient may also be 0. It fluctuates up and down, but the cumulative delay gradient should approach 0, so the change of the cumulative delay gradient is close to 0. Therefore, if you want to judge the bandwidth usage status based on the change of the cumulative delay gradient, the setting of the threshold is very important. If the threshold is Fixed value, setting too large may not detect network congestion, setting too small may be too sensitive, WebRTC uses an adaptive dynamic threshold method.

计算方式：threshold(t(i)) = threshold(t(i-1)) + k * [ t(i) - t(i-1) ] * [ | trend(t(i)) | - threshold(t(i-1)) ]

Where k represents the rate of change, when | trend(t(i)) | < threshold(t(i-1)), the value of k is 0.039, otherwise the value of k is 0.0087,

threshold(t(i)) means that we need to determine the new threshold after calculating the i-th package, threshold(t(i-1)) means the threshold determined after calculating the i-1th package, t(i) - t (i-1) indicates the time difference between the two packages calculating the delay gradient, and trend(t(i)) indicates the trend of the currently calculated delay gradient (the enlarged value).

（4）Rate controller

After judging the current usage state of the bandwidth, it is necessary to adjust the maximum bit rate value according to the current state, as shown in the figure below.

When it is in the overuse state, it corresponds to the Decr state. At this time, the maximum bit rate value should be reduced to 0.85 times the maximum acked_bitrate in the past 500ms time window. The acked_bitrate can feedback the packet reception status of the message through RTCP and combine with the locally maintained sending list get

When it is in the underuse state, corresponding to the Hold state, the current maximum bit rate should remain unchanged at this time

When it is in the normal state, corresponding to Incr, the code rate can be increased appropriately at this time, and the increase is 1.08 times of the original maximum code rate value

The code for the Rate Controller corresponding to WebRTC to adjust the maximum bit rate is as follows.

void AimdRateControl::ChangeBitrate(const RateControlInput& input,
                                    Timestamp at_time) {
  absl::optional<DataRate> new_bitrate;
  DataRate estimated_throughput =
      input.estimated_throughput.value_or(latest_estimated_throughput_);
  if (input.estimated_throughput)
    latest_estimated_throughput_ = *input.estimated_throughput;

  // An over-use should always trigger us to reduce the bitrate, even though
  // we have not yet established our first estimate. By acting on the over-use,
  // we will end up with a valid estimate.
  if (!bitrate_is_initialized_ &&
      input.bw_state != BandwidthUsage::kBwOverusing)
    return;

  ChangeState(input, at_time);

  // We limit the new bitrate based on the troughput to avoid unlimited bitrate
  // increases. We allow a bit more lag at very low rates to not too easily get
  // stuck if the encoder produces uneven outputs.
  const DataRate troughput_based_limit =
      1.5 * estimated_throughput + DataRate::KilobitsPerSec(10);

  switch (rate_control_state_) {
    case kRcHold:
      break;

    case kRcIncrease:
      if (estimated_throughput > link_capacity_.UpperBound())
        link_capacity_.Reset();

      // Do not increase the delay based estimate in alr since the estimator
      // will not be able to get transport feedback necessary to detect if
      // the new estimate is correct.
      // If we have previously increased above the limit (for instance due to
      // probing), we don't allow further changes.
      if (current_bitrate_ < troughput_based_limit &&
          !(send_side_ && in_alr_ && no_bitrate_increase_in_alr_)) {
        DataRate increased_bitrate = DataRate::MinusInfinity();
        if (link_capacity_.has_estimate()) {
          // The link_capacity estimate is reset if the measured throughput
          // is too far from the estimate. We can therefore assume that our
          // target rate is reasonably close to link capacity and use additive
          // increase.
          DataRate additive_increase =
              AdditiveRateIncrease(at_time, time_last_bitrate_change_);
          increased_bitrate = current_bitrate_ + additive_increase;
        } else {
          // If we don't have an estimate of the link capacity, use faster ramp
          // up to discover the capacity.
          DataRate multiplicative_increase = MultiplicativeRateIncrease(
              at_time, time_last_bitrate_change_, current_bitrate_);
          increased_bitrate = current_bitrate_ + multiplicative_increase;
        }
        new_bitrate = std::min(increased_bitrate, troughput_based_limit);
      }

      time_last_bitrate_change_ = at_time;
      break;

    case kRcDecrease: {
      DataRate decreased_bitrate = DataRate::PlusInfinity();

      // Set bit rate to something slightly lower than the measured throughput
      // to get rid of any self-induced delay.
      decreased_bitrate = estimated_throughput * beta_;
      if (decreased_bitrate > current_bitrate_ && !link_capacity_fix_) {
        // TODO(terelius): The link_capacity estimate may be based on old
        // throughput measurements. Relying on them may lead to unnecessary
        // BWE drops.
        if (link_capacity_.has_estimate()) {
          decreased_bitrate = beta_ * link_capacity_.estimate();
        }
      }
      if (estimate_bounded_backoff_ && network_estimate_) {
        decreased_bitrate = std::max(
            decreased_bitrate, network_estimate_->link_capacity_lower * beta_);
      }

      // Avoid increasing the rate when over-using.
      if (decreased_bitrate < current_bitrate_) {
        new_bitrate = decreased_bitrate;
      }

      if (bitrate_is_initialized_ && estimated_throughput < current_bitrate_) {
        if (!new_bitrate.has_value()) {
          last_decrease_ = DataRate::Zero();
        } else {
          last_decrease_ = current_bitrate_ - *new_bitrate;
        }
      }
      if (estimated_throughput < link_capacity_.LowerBound()) {
        // The current throughput is far from the estimated link capacity. Clear
        // the estimate to allow an immediate update in OnOveruseDetected.
        link_capacity_.Reset();
      }

      bitrate_is_initialized_ = true;
      link_capacity_.OnOveruseDetected(estimated_throughput);
      // Stay on hold until the pipes are cleared.
      rate_control_state_ = kRcHold;
      time_last_bitrate_change_ = at_time;
      time_last_bitrate_decrease_ = at_time;
      break;
    }
    default:
      assert(false);
  }

  current_bitrate_ = ClampBitrate(new_bitrate.value_or(current_bitrate_));
}

Finally, save the estimated maximum bit rate value based on the delay gradient to the delay_based_limit_ variable of SendSideBandwidthEstimation.

(2) Bandwidth estimation based on packet loss rate

The idea of bandwidth estimation based on packet loss at the sending end is to judge whether there is congestion based on the packet loss rate.

When the packet loss rate is greater than 10%, it is considered to be congested. At this time, the sending bit rate should be actively reduced to reduce congestion; when the packet loss rate is less than 2%, it is considered that the network is in good condition, and the sending bit rate can be appropriately increased to detect whether there are more available Bandwidth; when the packet loss rate is between 2% and 10%, it is considered that the network condition is normal, and at this time, keep the same sending bit rate as last time.

For the acquisition of the packet loss rate, the sender judges the packet loss rate by the number of lost packets of the RTCP RR message and the number of packets with the largest sequence number received. The format and field meanings of the RR message are as follows.

The code for WebRTC to receive RR packets and estimate the bandwidth based on the packet loss rate is as follows.

Finally, in UpdateTargetBitrate, the smaller value between the estimated value obtained according to packet loss and the estimated value obtained according to the delay gradient will be taken as the final estimated maximum bit rate.