Graphical DTS and PTS
There are two kinds of timestamps in FFmpeg: DTS (Decoding Time Stamp, decoding timestamp) and PTS (Presentation Time Stamp, displaying timestamp).
Since the video is encoded into I, B, P and other frames, as shown in the figure below:
Assuming that there are I, B, P frames now, what about transmission and display?
If the transmission is in the order of display: the transmission order
is I->B>P
After decoding the B frame, since the B frame cannot be displayed alone, it can only wait for the subsequent P frame
If the transmission is not in order, but in the decoding order:
the transmission order is I->P->B
No matter which method is used for transmission and display, once you have a B frame, you need to tell the other party when to display this frame,
so there are PTS and DTS, namely Presentation Time Stamp and Decode Time Stamp
PTS tells the other party when to display this frame, and DTS tells when to decode this frame
If there is no B frame, PTS and DTS are the same:
if there is B frame, PTS and DTS will be inconsistent:
for example, if the video is 25 frames per second, it is calculated in milliseconds, 1000/25=40ms, in the first Accumulate on frame pts.
Audio is accumulated on the same time base according to the sampling rate and number of samples, for example, 1024 samples, 44100 sampling rate, milliseconds.
1000*1024/44100=23.21995464852607709750566893424 ms
The basic theory is like this, but the actual synchronization is far from that simple. A series of disconnection, disconnection, weak network, frame loss, frame skipping, etc. will hinder your synchronization. Do synchronization in specific situations, and wait for the masters to give more robust synchronization measures.
fixed frame rate
1. Video Timestamp
pts = inc++ *(1000/fps); 其中inc是一个静态的，初始值为0，每次打完时间戳inc加1. 在ffmpeg，中的代码为 pkt.pts= m_nVideoTimeStamp++ * (m_VCtx->time_base.num * 1000 / m_VCtx->time_base.den);
2. Audio Timestamp
pts = inc++ * (frame_size * 1000 / sample_rate) 在ffmpeg中的代码为 pkt.pts= m_nAudioTimeStamp++*(m_ACtx>frame_size*1000/m_ACtx>sample_rate);
Sampling frequency refers to the number of times the amplitude samples of the sound wave are taken per second when digitizing an analog sound waveform. . The frequency range of normal human hearing is about 20Hz~20kHz. According to the Nyquist sampling theory, in order to ensure that the sound is not distorted, the sampling frequency should be around 40kHz. Commonly used audio sampling frequencies are 8kHz, 11.025kHz, 22.05kHz, 16kHz, 37.8kHz, 44.1kHz, 48kHz, etc. If a higher sampling frequency is used, the sound quality of DVD can also be achieved. Decode AAC audio with a sampling rate of 44.1kHz , the decoding time of one frame must be controlled within 23.22 milliseconds. Background knowledge:
(an AAC original frame contains 1024 samples and related data for a period of time) Analysis:
- The playback time of an AAC
audio frame = the number of sampling samples corresponding to one AAC frame/sampling frequency (unit is s) 1024 samples per frame. The sampling rate is Samplerate 44100KHz, 44100 samples per second, so according to the formula, the playback time of an audio frame = the number of sampling samples corresponding to one AAC frame/sampling frequency. The current playback time of one AAC frame is = 1024*1000000/44100= 22.32ms (unit is ms) - MP3
Each frame of mp3 is 1152 bytes, then:
frame_duration = 1152 * 1000000 / sample_rate
For example: when sample_rate = 44100HZ, the calculated duration is 26.122ms, which is the often heard mp3 playback time of each frame is fixed at 26ms origin.
2. There are a lot of capture cards and cameras with variable frame rate. When doing capture, it is clearly set to 25FPS, but when the actual collected data is called back, it is found that the interval is not 40 milliseconds, but 50, 60, or even 100. time interval. This brings great difficulty to time stamping after encoding.
Reference article:
link: https://www.cnblogs.com/linyilong3/p/9940230.html .
Relationship between I, P, B frames and PTS, DTS: https://www.cnblogs.com/qingquan/archive/ 2011/07/27/2118967.html .
Explanation of DTS and PTS: https://blog.csdn.net/ai2000ai/article/details/77367481 .