Overview

To play a video file on the Internet, a video player needs to go through the following steps: de-protocol, de-encapsulate, decode video and audio, and synchronize video and audio. If you play local files, you don't need to de-protocol, and it involves the following steps: de-encapsulation, decoding of video and audio, and video and audio synchronization. Their process is shown in the figure:
Insert picture description here

The function of the solution protocol is to parse the data of the streaming media protocol into the corresponding standard encapsulation format data. When video and audio are disseminated on the Internet, various streaming media protocols, such as HTTP, RTMP, or MMS, are often used. While these protocols transmit video and audio data, they also transmit some signaling data. These signaling data include control of playback (play, pause, stop), or description of network status, etc. In the process of de-agreement, the signaling data will be removed and only the video and audio data will be retained. For example, the data transmitted using the RTMP protocol will output data in FLV format after de-protocol operation.

The function of decapsulation is to separate the input data in the encapsulated format into audio stream compression coded data and video stream compression coded data. There are many types of encapsulation formats, such as MP4, MKV, RMVB, TS, FLV, AVI, etc. Its function is to put the compressed and encoded video data and audio data together in a certain format. For example, FLV format data, after decapsulation operation, output H.264 encoded video stream and AAC encoded audio stream.

The function of decoding is to compress and encode video/audio data into uncompressed video/audio raw data. Audio compression coding standards include AAC, MP3, AC-3, etc., video compression coding standards include H.264, MPEG2, VC-1, etc. Decoding is the most important and most complex link in the entire system. Through decoding, the compressed and encoded video data is output as uncompressed color data, such as YUV420P, RGB, etc.; the compressed and encoded audio data is output as uncompressed audio sample data, such as PCM data.

The function of video and audio synchronization is to synchronize the decoded video and audio data according to the parameter information obtained during the processing of the decapsulation module, and send the video and audio data to the graphics card and sound card of the system for playback.

Insert picture description here