Analysis of video capture in webrtc (2) Distribution of video frames

need

In the previous article, the interface of the video capture module in webrtc was introduced. After the video capture module throws video data, the processing requirements for video frames generally include the following:

1. The video frame is sent to the encoding module for encoding
2. The video frame is sent to the rendering module for local echo
3. Video frames are preprocessed (such as adding watermarks, subtitles), and then sent to echo or encode
4. Video frames will be sent to multiple encoders to generate multiple code streams with different resolutions and bit rates

It is equivalent to a "source" having multiple "tributaries". The video frame data obtained through the callback is the "source", and the various requirements above are the "tributaries". Video frame data should be distributed on demand to meet the needs of each "tributary". Then it is reflected in the code as one source and multiple sinks.

The distribution framework involves classes

There is a distribution framework for video frames in webrtc, which can be directly multiplexed, including videoSourceInterface, videoSinkInterface,VideoBroadcaster

videoSourceInterface

As the base class of source, it refers to "source"

• The main interfaces are AddOrUpdateSink()and RemoveSink(), videoSourceInterfacemultiple sinks can be added
• AddOrUpdateSinkThe second parameter of the interface is VideoSinkWantsa structure, the following is its definition

struct VideoSinkWants {
  VideoSinkWants();
  VideoSinkWants(const VideoSinkWants&);
  ~VideoSinkWants();
  // Tells the source whether the sink wants frames with rotation applied.
  // By default, any rotation must be applied by the sink.
  bool rotation_applied = false;

  // Tells the source that the sink only wants black frames.
  bool black_frames = false;

  // Tells the source the maximum number of pixels the sink wants.
  int max_pixel_count = std::numeric_limits<int>::max();
  // Tells the source the desired number of pixels the sinks wants. This will
  // typically be used when stepping the resolution up again when conditions
  // have improved after an earlier downgrade. The source should select the
  // closest resolution to this pixel count, but if max_pixel_count is set, it
  // still sets the absolute upper bound.
  absl::optional<int> target_pixel_count;
  // Tells the source the maximum framerate the sink wants.
  int max_framerate_fps = std::numeric_limits<int>::max();

  // Tells the source that the sink wants width and height of the video frames
  // to be divisible by |resolution_alignment|.
  // For example: With I420, this value would be a multiple of 2.
  // Note that this field is unrelated to any horizontal or vertical stride
  // requirements the encoder has on the incoming video frame buffers.
  int resolution_alignment = 1;
};

The main thing is to describe the sink's expected resolution, frame rate, etc., and the above comments are also very clear.

videoSinkInterface

The main interface as the base class of sink
is OnFrame()andOnDiscardeFrame()

These two classes represent source and sink. Multiple sinks can be added to a source. The source transfers the video frame data to the sink by calling the OnFrame method in the sink.

VideoBroadcaster video frame distribution class

-Class VideoBroadcaster, you can tell by the name that it is a video frame distribution class. As you can see from the above class diagram, VideoSourceBaseit is both a source and a sink.

• VideoBroadcasterIt can be placed in a pair of sinks, but it will merge the wants of all sink sets. The frame rate value is the minimum value in the sink want, and the resolution value is also the minimum value in the sink want.

• However, when the video frame provided by the source does not satisfy the sink, the video frame data needs to be adapted, so VideoAdpaterthis is the effect

video adapter

VideoAdpaterIt is equivalent to a filter, which filters the video frames in the source to meet the requirements of the sink.
VideoAdpaterIt can filter the frame rate and scale the resolution.

basic framework

Through the relationship between the above classes, a framework for video frame distribution is spelled out

videocaputreRepresents the video acquisition module, which represents the "source" of the total data, sends the video frame data through the callback VideoBroadcaster, VideoBroadcasterand then distributes the video frame to each sink.

In the previous article, VideoCaptureModulethere is a callback registration interface void RegisterCaptureDataCallback(rtc::VideoSinkInterface<VideoFrame>* dataCallback), which can be VideoBroadcasterregistered as a sink.

It should be noted that it VideoBoradcasterdoes not contain VideoAdpaterclasses, it has no adaptation function, it is simply distributed. If you want to adapt the video frame data, you need to VideoBoradcasteradd it before or after it VideoAdpater.

Variety

The usage in webrtc VideoBoradcasteris limited, it cannot realize the distribution of different frame rates and different resolutions (it is mentioned above that it will merge all sink wants, take the frame rate in sink wants, and the minimum resolution ). It is suitable for the distribution of the purpose of the frame, for example, some are sent for encoding, and some are sent for local echo, all at the same resolution and frame rate.
Therefore, the combination diagram above cannot satisfy the scenario of generating multiple streams with different resolutions and frame rates.

Consider such a requirement:

1. Can generate three streams, they have different resolutions, frame rates
2. can echo locally

So it needs to be combined in the following way.

The video captured by videocatpure is 30fps, 720P. It is necessary to generate three code streams (with three encoder objects) and local echo, which VideoAaptercannot be placed in VideoBroadcasterfront, because VideoBroadcasteronly the same frame rate and resolution can be output. One needs to be placed in front of the video encoder VideoAapterto meet the frame rate and resolution requirements.

VideoAapterWhen the frame rate is met, frame loss processing will be performed, so these three VideoAapter+Video encodershould be in one thread respectively.

Example in webrtc

In the implementation of the example , and are integrated peer_conection_clientthrough the class , you can refer to how to connect them. However, in the implementation of the browser, it should not be implemented, because its implementation cannot meet the needs of multiple code streams with different resolutions and frame rates.TestVideoCapturerVideoAdapterVideoBroadcasterTestVideoCapturer