Hi everyone, and welcome to the Stop Refactoring channel.
From this issue, we officially enter the introduction of audio and video processing.
In this issue, we discuss the conversion of audio and video files , such as converting MP4 to AVI, MP4 to RTMP, etc.
The code mentioned in the content will be placed on GitHub , interested friends can go to GitHub to download.
We discuss in this order:
1. The role of video encapsulation
2. The working principle of transcapsulation
3. Transpack into a video file
4. Transpack into a live stream
The role of video packaging
As discussed in the previous "Working Principles of Audio and Video Transcoding", a video file is essentially divided into three layers, encapsulation, encoding, and basic data .
The function of the encapsulation format , simply put, is to organize data such as audio data, video data, and basic information into files according to certain structural rules .
Although the data arrangement will not affect the audio and video data itself , it will affect some specific scenarios.
For example, video files generally use MP4, FLV and other encapsulation formats, and live streams use protocols such as RTMP and HTTP-FLV.
For detailed descriptions of video packaging formats such as MP4, FLV, and HLS, please refer to the previous issue of "Video Format".
For detailed descriptions of streaming media protocols such as RTMP, HTTP-FLV, and HLS, please refer to the previous issue of "Live Streaming Protocol".
How Repackaging Works
There are many application scenarios for transcapsulation, such as converting MP4 files to FLV files, converting MP4 files to RTMP live streams, and converting RTMP live streams to MP4 files.
It should be noted here that transcapsulation only changes the arrangement of file data . If you need to change the resolution, bit rate and other settings, you need to transcode to complete.
For the difference between transcapsulation and transcoding, please refer to the previous issue of "Working Principles of Audio and Video Transcoding".
The working principle of transcapsulation is very simple, that is, after decapsulating the video file or video stream , repackage it into a video file or video stream according to the new format.
When transcapsulating, audio data and video data do not need to be processed differently . This process is streaming, that is, only a part of the data is processed at a time, and the process is repeated until the processing is completed.
In addition, if the video file is converted to a live stream , a time interval needs to be added during the loop process , otherwise an error will be reported due to the fast streaming.
Convert to video file
The following is the specific FFmpeg operation process, and the corresponding sample code will also be explained synchronously.
Transcapsulate into video files , such as convert MP4 files into FLV files, the corresponding sample code is remux_tofile.cpp . After the environment is built, run the program as shown in the figure:
The code flow is 4 steps.
The first step is to open the source file and obtain the source file information;
The second step is to construct the output file . Since the transcapsulation does not need to process the audio and video data, the output file can be constructed directly according to the track information of the source file;
The third step is to process the data , decapsulate the source data in a loop, and write the package to the output file until the source data is read;
The fourth step is to close the input and output files . After the processing is completed, you need to write the tail information to the output file before closing, otherwise there may be problems with the video file.
If the live stream is converted into a video file, the above code is also applicable and does not need to be changed, because the decapsulated API will still return a result less than 0 when the live stream is interrupted or ended.
If you want to interrupt the live stream recording in advance, you need to find another time to write tail information to the output file.
Transpack into live stream
Transcapsulate into a live stream .
For example, to convert an MP4 file into an RTMP stream, the corresponding sample code is remux_tostream.cpp , and the running program is shown in the figure:
The overall code is similar to the code for transcapsulating into a video file , but a time interval is added here to prevent errors caused by rewinding too quickly.
What needs to be explained here is that the decapsulated packet data contains dts decoding timestamp and pts playback timestamp.
We generally use dts as the basis for calculating the time interval . Of course, dts needs to be multiplied by timebase to be the real time, and timebase is the timebase of the corresponding track of the source video file.
Generally, dts and pts are equal , but in the case of H264 b frames, etc., pts may be greater than dts , so if pts is used as the basis for calculating the time interval, the live stream may freeze.
Our example MP4 file has no b-frame, so every dts and pts are equal.
In the actual recording and broadcasting scenario , that is, when the video file is transferred to the live streaming scenario, the video file should not have settings such as b frames that make dts and pts unequal, because even if the program calculates the time interval according to dts, if the pts of a certain frame Too far away from dts will also cause live broadcast lag.
If the source file cannot eliminate the b frame , then simple transcapsulation cannot guarantee the smoothness of the live stream , and the transcoding process needs to be added. However, transcoding needs to consume more performance. If the transcoding process is added just to eliminate b frames, the gain outweighs the gain.
Summarize
The code of this issue has been uploaded to Github, friends who need it can download it, and there are instructions for building the compilation and running environment.
If you are confused about the installation of FFmpeg environment, you can refer to the previous video "FFmpeg". Of course, you can also choose the docker container we made.