Custom Player Series
Chapter 1 Video rendering
Chapter 2 Audio (push) playback
Chapter 3 Audio (pull) playback
Chapter 4 Clock synchronization (this chapter)
Chapter 5 Universal clock synchronization
Chapter 6 Player
Article directory
foreword
After using ffmpeg and sdl to play video, it is necessary to achieve clock synchronization to play video normally, especially in the case of audio, we usually need to synchronize video to audio to ensure the synchronization of audio and video. The clock synchronization of video is sometimes difficult to understand, and even knowing the theory cannot ensure the realization, and it is necessary to obtain various parameters and specific implementation logic through practice. This article will introduce some specific implementations of video clock synchronization.
1. Direct delay
We can directly delay the time when we play the video. This method is not accurate, but it is also a rudimentary method.
1. Delay based on frame rate
A fixed delay is performed every time a frame is rendered, and the delay time is calculated by the frame rate.
//获取视频帧率
AVRational framerate = play->formatContext->streams[video->decoder.streamIndex]->avg_frame_rate;
//根据帧率计算出一帧延时
double duration = (double)framerate.num / framerate.den;
//显示视频
略
//延时
av_usleep(duration* 1000000);
2. Delay according to duration
Each rendered frame is delayed according to its duration, which is usually included in the video packaging format.
//获取当前帧的持续时间，下列方法有可能会无法获取duration，还有其他方法获取duration这里不具体说明，比如ffplay通过缓存多帧，来计算duartion。或者自己实现多帧估算duration。
AVRational timebase = play->formatContext->streams[video->decoder.streamIndex]->time_base;
double duration = frame->pkt_duration * (double)timebase.num / timebase.den;
//显示视频
略
//延时
av_usleep(duration* 1000000);
Second, synchronize to the clock
Note: This part only talks about video clock synchronization and does not involve audio.
The above simple delay can meet the playback requirements to a certain extent, but there are also problems, and the accuracy of the time cannot be guaranteed. In particular, demultiplexing and decoding will also consume time, and a simple fixed delay will lead to cumulative delay. At this time, we need a clock to calibrate the playback time of each frame.
1. Synchronize to an absolute clock
The simpler way is to synchronize to the clock in an absolute way, that is, how long the video is, it must be played according to the length of the system time after it starts, and if the video is slow, it will continue to drop frames until it catches up with the time.
Define a video start time, outside of the playback loop.
//视频起始时间，单位为秒
double videoStartTime=0;
Clock synchronization in playback loop. The frame of the following code is the decoded AVFrame.
//以下变量时间单位为s
//当前时间
double currentTime = av_gettime_relative() / 1000000.0 - videoStartTime;
//视频帧的时间
double pts = frame->pts * (double)timebase.num / timebase.den;
//计算时间差,大于0则late，小于0则early。
double diff = currentTime - pts;
//视频帧的持续时间，下列方法有可能会无法获取duration，还有其他方法获取duration这里不具体说明，比如ffplay通过缓存多帧，来计算duartion。
double duration = frame->pkt_duration * (double)timebase.num / timebase.den;
//大于阈值，修正时间，时钟和视频帧偏差超过0.1s时重新设置起点时间。
if (diff > 0.1)
{
videoStartTime = av_gettime_relative() / 1000000.0 - pts;
currentTime = pts;
diff = 0;
}
//时间早了延时
if (diff < 0)
{
//小于阈值,修正延时，避免延时过大导致程序卡死
if (diff< -0.1)
{
diff =-0.1;
}
av_usleep(-diff * 1000000);
currentTime = av_gettime_relative() / 1000000.0 -videoStartTime;
diff = currentTime - pts;
}
//时间晚了丢帧，duration为一帧的持续时间，在一个duration内是正常时间，加一个duration作为阈值来判断丢帧。
if (diff > 2 * duration)
{
av_frame_unref(frame);
av_frame_free(&frame);
//此处返回即不渲染，进行丢帧。也可以渲染追帧。
return;
}
//显示视频
略
2. Synchronize to video clock
Synchronizing to the video clock is to update the video clock according to the pts of the current frame every time rendering is based on the pts of the video playback. The difference with the above is only the bottom line of clock update code.
//更新视频时钟
videoStartTime = av_gettime_relative() / 1000000.0 - pts;
Because it is basically the same as the code in the previous section, no specific instructions are given, just refer to the instructions in the previous section.
//以下变量时间单位为s
//当前时间
double currentTime = av_gettime_relative() / 1000000.0 - videoStartTime;
//视频帧的时间
double pts = frame->pts * (double)timebase.num / timebase.den;
//计算时间差,大于0则late，小于0则early。
double diff = currentTime - pts;
//视频帧的持续时间，下列方法有可能会无法获取duration，还有其他方法获取duration这里不具体说明，比如ffplay通过缓存多帧，来计算duartion。
double duration = frame->pkt_duration * (double)timebase.num / timebase.den;
//大于阈值，修正时间，时钟和视频帧偏差超过0.1s时重新设置起点时间。
if (diff > 0.1)
{
videoStartTime = av_gettime_relative() / 1000000.0 - pts;
currentTime = pts;
diff = 0;
}
//时间早了延时
if (diff < 0)
{
//小于阈值,修正延时，避免延时过大导致程序卡死
if (diff< -0.1)
{
diff =-0.1;
}
av_usleep(-diff * 1000000);
currentTime = av_gettime_relative() / 1000000.0 - videoStartTime;
diff = currentTime - pts;
}
//时间晚了丢帧，duration为一帧的持续时间，在一个duration内是正常时间，加一个duration作为阈值来判断丢帧。
if (diff > 2 * duration)
{
av_frame_unref(frame);
av_frame_free(&frame);
//此处返回即不渲染，进行丢帧。也可以渲染追帧。
return;
}
//更新视频时钟
videoStartTime = av_gettime_relative() / 1000000.0 - pts;
//显示视频
略
3. Sync to Audio
1. Calculation of audio clock
To synchronize to the audio, we must first calculate the audio clock, and the pts can be calculated through the data length of the audio playback.
Define two variables, the pts of the audio, and the start time of the audio clock startTime.
//下列变量单位为秒
double audioPts=0;
double audioStartTime=0;
Calculate the audio clock in sdl's audio playback callback. Where spec is SDL_AudioSpec is the fourth parameter of SDL_OpenAudioDevice.
//音频设备播放回调
static void play_audio_callback(void* userdata, uint8_t* stream, int len) {
//写入设备的音频数据长度
int dataSize;
//将数据拷贝到stream
略
//计算音频时钟
if (dataSize > 0)
{
//计算当前pts
audioPts+=(double) (dataSize)*/ (spec.freq * av_get_bytes_per_sample(forceFormat) * spec.channels);
//更新音频时钟
audioStartTime= = av_gettime_relative() / 1000000.0 -audioPts;
}
}
2. Sync to audio clock
After having the audio clock, we need to synchronize the video to the audio, and add synchronization logic on the basis of 2 and 2.
//同步到音频
double avDiff = 0;
avDiff = videoStartTime - audioStartTime;
diff += avDiff;
full code
//以下变量时间单位为s
//当前时间
double currentTime = av_gettime_relative() / 1000000.0 - videoStartTime;
//视频帧的时间
double pts = frame->pts * (double)timebase.num / timebase.den;
//计算时间差,大于0则late，小于0则early。
double diff = currentTime - pts;
//视频帧的持续时间，下列方法有可能会无法获取duration，还有其他方法获取duration这里不具体说明，比如ffplay通过缓存多帧，来计算duartion。
double duration = frame->pkt_duration * (double)timebase.num / timebase.den;
//同步到音频
double avDiff = 0;
avDiff = videoStartTime - audioStartTime;
diff += avDiff;
//大于阈值，修正时间，时钟和视频帧偏差超过0.1s时重新设置起点时间。
if (diff > 0.1)
{
videoStartTime = av_gettime_relative() / 1000000.0 - pts;
currentTime = pts;
diff = 0;
}
//时间早了延时
if (diff < 0)
{
//小于阈值,修正延时，避免延时过大导致程序卡死
if (diff< -0.1)
{
diff =-0.1;
}
av_usleep(-diff * 1000000);
currentTime = av_gettime_relative() / 1000000.0 - videoStartTime;
diff = currentTime - pts;
}
//时间晚了丢帧，duration为一帧的持续时间，在一个duration内是正常时间，加一个duration作为阈值来判断丢帧。
if (diff > 2 * duration)
{
av_frame_unref(frame);
av_frame_free(&frame);
//此处返回即不渲染，进行丢帧。也可以渲染追帧。
return;
}
//更新视频时钟
videoStartTime = av_gettime_relative() / 1000000.0 - pts;
//显示视频
略
Summarize
That's what I'm going to talk about today. This article briefly introduces several methods of video clock synchronization. It's not particularly difficult, but there are relatively few information found on the Internet. The implementation of ffplay that can be referred to is also a bit complicated. The implementation of this article draws on ffplay. The clock synchronization implemented in this article can still be optimized, such as using pid for dynamic control. And the calculation of duration can be fine-tuned.