I. Introduction
I use ffmpeg for audio and video synchronization. I personally think that this is the most difficult one in the basic processing of ffmpeg. Countless people are stuck here. No way. I have tried various demos on the Internet, and they are basically scum, or It only supports a very small number of video files. For example, the received data packets are one frame of video and one frame of audio. Either it is impossible to synchronize at all, or the progress jumps over and crashes directly. In fact, the most perfect audio and video The synchronization demo is ffplay. I have personally tested dozens of various audio and video local files, dozens of video stream files, and they are all perfect. Of course, this is my own, and I'm not perfect.
If you only play the video stream (without audio stream), you may not need to synchronize audio and video, so when you only play the rtsp video stream at the beginning, you did not consider the synchronization problem at all, because you did not encounter it and you don’t need it. When I planted video streams like rtmp, http, m3u8, the problem got bigger. It came from the hls format video stream files at one time, and small video files came over. If they are not synchronized, it means suddenly There are a lot of pictures that have been swiped in the past, and the next time they come, they will be swiped again. This requires you to calculate the synchronization yourself. The data packets received last time are put in the queue and displayed when they need to be displayed.
Common audio and video synchronization methods:
- Controlled by fps, fps indicates how many frames are played in one second, such as 25 frames. You can calculate the time used to decode one frame by yourself. One frame occupies (1000/25=40 milliseconds). It is processed by delay. This is actually The most scumbag way.
- Remember the time startTime to start decoding, and calculate the pts time through av_rescale_q. The difference between the two is the time that needs to be delayed. Call av_usleep to delay. Only part of the file is normal, but it is not normal in many cases.
- The audio is synchronized to the video, and the video clock is used as the master clock. I haven't tried it. Many people on the Internet say this method is not good.
- The video is synchronized to the audio, and the audio clock is used as the master clock. I haven't tried it. It is said that most people use this method.
- The audio and video are synchronized to the external clock, and the external clock is used as the master clock. The final method is easy to understand and does not interfere with each other, and each synchronizes itself according to the external clock.
- ffplay has built-in three synchronization strategies, which can be controlled by parameters. The default is to synchronize video to audio.
2. Features
- Multi-threaded real-time playback of video stream + local video + USB camera, etc.
- Support windows+linux+mac, support ffmpeg3 and ffmpeg4, support 32-bit and 64-bit.
- Multi-threaded display images, not stuck in the main interface.
- Reconnect the webcam automatically.
- You can set the border size, offset and border color.
- You can set whether to draw OSD labels, that is, label text or pictures and label positions.
- Two OSD positions and styles can be set.
- Can set whether to save to file and file name.
- You can directly drag the file to the ffmpegwidget control to play.
- Support common video streams such as h265 video stream + rtmp.
- Can pause and resume playing.
- Support storage of single video files and timing storage of video files.
- Customize the top floating bar, send a click signal notification, and set whether to enable it.
- You can set the screen stretch fill or equal proportion fill.
- You can set the decoding to be speed priority, quality priority, and equalization processing.
- Can take screenshots (original pictures) and screenshots of videos.
- Video file storage supports bare stream and MP4 files.
- Audio and video are perfectly synchronized, using an external clock synchronization strategy.
- Support seek to locate the playback position.
- Supports hard decoding such as qsv, dxva2, d3d11va, etc.
- Support OpenGL to draw video data, very low CPU usage.
- Support Android and embedded linux, just cross compile.
Three, renderings
Four, related sites
- Domestic site: https://gitee.com/feiyangqingyun/QWidgetDemo
- International site: https://github.com/feiyangqingyun/QWidgetDemo
- Personal homepage: https://blog.csdn.net/feiyangqingyun
- Zhihu Homepage: https://www.zhihu.com/people/feiyangqingyun/
- Experience address: https://blog.csdn.net/feiyangqingyun/article/details/97565652
Five, the core code
void FFmpegSync::run()
{
reset();
while (!stopped) {
//暂停状态或者队列中没有帧则不处理
if (!thread->isPause && packets.count() > 0) {
mutex.lock();
AVPacket *packet = packets.first();
mutex.unlock();
//h264的裸流文件同步有问题,获取不到pts和dts,暂时用最蠢的办法延时解决
if (thread->formatName == "h264") {
int sleepTime = (1000 / thread->videoFps) - 5;
msleep(sleepTime);
}
//计算当前帧显示时间 外部时钟同步
ptsTime = getPtsTime(thread->formatCtx, packet);
if (!this->checkPtsTime()) {
msleep(1);
continue;
}
//显示当前的播放进度
checkShowTime();
//0-表示音频 1-表示视频
if (type == 0) {
thread->decodeAudio(packet);
} else if (type == 1) {
thread->decodeVideo(packet);
}
//释放资源并移除
thread->free(packet);
mutex.lock();
packets.removeFirst();
mutex.unlock();
}
msleep(1);
}
clear();
stopped = false;
}
bool FFmpegSync::checkPtsTime()
{
bool ok = false;
if (ptsTime > 0) {
if (ptsTime > offsetTime + 100000) {
bufferTime = ptsTime - offsetTime + 100000;
}
int offset = (type == 0 ? 1000 : 5000);
offsetTime = av_gettime() - startTime + bufferTime;
if ((offsetTime <= ptsTime && ptsTime - offsetTime <= offset) || (offsetTime > ptsTime)) {
ok = true;
}
} else {
ok = true;
}
return ok;
}