Detailed explanation of SGPlayer principle
SGPlayer is a media resource player framework based on AVPlayer and FFmpeg. Support panoramic video, RTMP, RTSP and other live streams; support iOS, macOS, tvOS three platforms at the same time. This article will introduce the implementation principle of key modules to you in the form of diagram + description.
- Project address: OSChina - SGPlayer
Initiation reason
Regarding video playback, the AVPlayer provided by Apple has excellent performance. When there is no special demand and the resources are controllable, it must be the first choice. But with the rise of VR and live streaming, AVPlayer alone is often not enough. For performance reasons, AVPlayer cannot be completely abandoned. After all, it has obvious advantages in on-demand. In the existing open source projects, the general positioning is relatively single, and it cannot take into account AVPlayer, live broadcast, and VR. In this way, three players need to be used at the same time to meet the needs, that is, AVPlayer is used for on-demand, an independent player is used for live broadcast, and an independent player is used for VR. Dealing with 3 different sets of interfaces and callback events in this way is really frustrating! The emergence of SGPlayer greatly simplifies this process.
Composition structure and playback process
The above figure shows the playback process and main components of SGPlayer. The following briefly introduces the division of labor of each component in the figure
SGPlayer
SGPlayer is an abstract player shell, it does not have the playback function itself. Only as a carrier for interaction with the outside world. The real playback is done by the internal SGAVPlayer and SGFFPlayer. The picture drawing is done by the internal SGDisplayView.
SGPlayerDecoder
SGPlayerDecoder is the selector of the playback kernel. It dynamically selects SGAVPlayer or SGFFPlayer for playback according to the resource type. You can customize the selection strategy of the playback kernel by changing its configuration parameters.
SGAVPlayer
SGAVPlayer is packaged based on AVPlayer, and the video image is output to SGDsiplayView and displayed according to the video type (panorama or plane). Audio is handled by the system without additional action.
SGFFPlayer
SGFFPlayer is packaged based on FFmpeg and supports nearly all mainstream video formats. The video image is also output to SGDsiplayView. Audio is output to SGAudioManager, and then SGAudioManager uses Audio Unit to play.
SGDisplayView
SGDisplayView is responsible for the drawing of the video screen. It does not draw video images by itself, it is only used as the parent view of the drawing layer. The real drawing is done by the internal AVPlayerLayer and SGGLViewController. The selection rules are shown in the following table.
| flat | panoramic | |
|---|---|---|
| SGAVPlayer | AVPlayerLayer | SGGLViewController |
| SGFFPlayer | SGGLViewController | SGGLViewController |
SGAudioManager
SGAudioManager is responsible for sound playback and audio event handling. Internally, AUGraph is used to do a layer of mixing. Through mixing, you can set the output volume of the sound and other operations.
summary
After understanding the functions of each component, reorganize the complete playback process
- SGPlayer received a playback request.
- Distributed by SGPlayerDecoder to SGAVPlayer or SGFFPlayer for playback according to the resource type.
- If using SGAVPlayer to play, output the picture to AVPlayerLayer or SGGLViewController in SGDisplayView according to the video type.
- If you use SGFFPlayer to play, output the video image to SGDisplayView, and output the audio to SGAudioManager.
The SGAVPlayer and SGFFPlayer that are really responsible for playback are shielded through the abstract SGPlayer, which ensures that no matter what type of resources, only a unified set of interfaces and callbacks are exposed to the outside world, and the differences between the playback cores are digested internally and the use cost is reduced as much as possible .
Panoramic image principle
Panoramic images and flat images are essentially a 2D image, the difference lies in the display carrier. For a plan view, the model used for display is a rectangle, and it is only necessary to map the pixels on the image to the rectangle one-to-one; while the model displayed by the panoramic image is a sphere, and every pixel on the image needs to be corresponding to the corresponding position on the sphere. There is not much difference between the two in the drawing process, only the mapping rules and rendering methods are slightly different.
Texture Rules
The process of attaching a flat image to a sphere is very similar to that of a globe. Take the above picture as an example, each pixel in the left picture can find its corresponding position on the right spherical surface. A key correspondence is listed below.
- All the points on the straight line AB correspond to the point J, and similarly all the points on the straight line CD correspond to the point K.
- The points on the straight line MN correspond one-to-one with the points on the equator.
- The points on the line AC/BD correspond one-to-one with the points on the front half of the green meridian.
- The points on the line EF correspond one-to-one with the points on the rear half of the green meridian.
ways of presenting
The above figure shows how the panoramic image is presented. Different from the plane, the panoramic image needs to place the viewing point at the center of the sphere, and stand at the center of the sphere to watch the image on the spherical surface. Finally, the projection of the surface ABCD on the plane ABCD is displayed on the screen.
summary
This part of the content involves a lot of OpenGL content in the implementation, and needs to have some OpenGL foundation. In binocular mode, distortion correction and dispersion correction are also required to ensure that the picture is truly restored. For the specific implementation, see SGGLViewController.
SGFFPlayer operation process
The above figure shows the collaboration flow chart of SGFFPlayer. The following is a brief introduction to the components in the figure.
threading model
There are a total of 4 threads in SGFFPlayer. Corresponds to the 4 blue circles in the figure.
- Data Read - Read Packet Loop
- Video Decode - Video Decode Loop
- Video Drawing - Video Display Loop
- Audio Playback - Audio Playback Loop
The figure hides the control condition of the thread. The entire playback process is completed with the cooperation of 4 threads.
SGVideoDecoder
SGVideoDecoder is a video decoder. It can be configured with synchronous and asynchronous decoding and whether to enable hard decoding during initialization. The above figure uses asynchronous decoding, and the default decoding thread correspondence is shown in the following table.
| flat | panoramic | |
|---|---|---|
| software decoding | asynchronous | Synchronize |
| hardware decoding | asynchronous | asynchronous |
- Synchronous decoding decodes the video packet immediately after receiving it and stores it in the video frame queue.
- Asynchronous decoding only stores the audio packet queue after receiving the video packet. When the independent decoding thread takes out the audio packet and completes the decoding, it stores it in the video frame queue.
SGAudioDecoder
SGAudioDecoder is an audio decoder, which adopts synchronous decoding, decodes the audio packet immediately after receiving it, and stores it in the audio frame queue.
Data queue SGFFPacketQueue, SGFFFrameQueue
- SGFFPacketQueue is a packet queue that manages packets before decoding (AVPacket).
- SGFFFrameQueue is a frame queue that manages decoded frames (SGFFVideoFrame or SGFFAudioFrame).
They all support synchronous and asynchronous acquisition of data, and synchronous acquisition is achieved through condition variables (NSCondition). When there is not enough data in the queue, the current thread is blocked, and the thread will not be woken up until a new element is added to the queue.
Frame multiplexing pool SGFFFramePool
This part is not shown in the above figure, but it can avoid some unnecessary performance overhead. Due to the large number of audio frames and video frames, 1 minute of video contains thousands of frames of data. If each frame is newly created, it will cause unnecessary waste of resources. The SGFFFrame created by SGFFFramePool will not be released immediately after use, but will be recycled by the reuse pool for next use, so as to achieve the purpose of creating only a minimum number of frame objects.
Audio and video synchronization
There are 3 kinds of commonly used synchronization at that time
- audio clock
- video clock
- homemade clock
In SGFFPlayer, the audio clock is used first, and when there is no audio track in the video, the video clock will be used for synchronization.
summary
Understand the functions of each component, take video asynchronous decoding as an example, and reorganize the whole process
- The data reading thread reads the data packets and distributes them to the audio decoder or video decoder according to the type of the data packets.
- If it is an audio packet, the audio decoder decodes the audio packet at the same time, and stores the decoded audio frame in the audio frame queue.
- If it is a video packet, because the video decoder performs asynchronous decoding, it only puts the video packet into the video packet queue, and waits for the video decoding thread to fetch the video packet from the queue.
- The video decoding thread loops out video packets from the video packet queue, decodes them at the same time, and stores the decoded video frames in the video frame queue.
- The audio playback thread loops out audio frames from the audio frame queue and plays them.
- The video presentation thread loops out video frames from the video frame queue and draws them.
At this point, the operation process of SGFFPlayer is very clear, and the playback function can be completed only by adding the corresponding conditional control in each link.
Summarize
The principle of SGPlayer is explained here. Since this article is mainly based on theory, there is no code posted. Interested students can find all the code implementations on OSChina . Hope to be helpful to everyone.