FFmpeg basic learning route

Introduction to ffmpeg

FFmpeg is a free, open-source, cross-platform audio and video processing tool designed to provide high-quality, high-speed multimedia processing. FFmpeg is designed to stream audio and video.

FFmpeg has become one of the most powerful and comprehensive audio and video processing tools recognized in the industry, and has been widely used. Most audio and video related applications and tool libraries directly or indirectly refer to FFmpeg.

The development of FFmpeg began in 2000, initially initiated by French programmer Fabrice Bellard, and later more and more open source contributors participated. Currently FFmpeg is maintained and developed by a team of several developers.

Audio and video development requires mastering the basics of images, video, and audio, and learning how to collect, render, process, and transmit them and develop and apply them.

Collection solves the problem of where the data comes from, rendering solves the problem of how the data is new, processing solves the problem of how the data is processed, and transmission solves the problem of how the data is shared. Every category here can be dug deep, and each topic is full of technical challenges.

1. Basic audio and video

1.1 Audio Basics

How to collect sound - the principle of analog-to-digital conversion
Why high-quality audio sampling rate>=44.1Khz
What is PCM
How many bits are used to represent a sampling point
Whether the sampled value is expressed as an integer or a floating point number
The volume has a great relationship with the sampling value
How many sampling points are used as a frame of data
How to arrange the sampling data of the left and right channels
What is PCM (Pulse Code Modulation)
Principles of Audio Coding

1.2 Video Basics

RGB color principle
Why do you need YUV format
what is a pixel
Resolution, frame rate, bit rate
Differences in YUV data storage formats
YUV memory alignment issue
Why does the screen display a green screen
H264 encoding principle
H264 IPB frame relationship

1.3 Basic knowledge of demultiplexing

What is demultiplexing, such as MP4 format
Why do you need different multiplexing formats MP4/FLV/TS
Common multiplexing format MP4/FLV/TS

1.4 FFmpeg6.0 development environment construction

Three major platforms: Windows, Ubuntu, and MAC
QT installation
FFmpeg command line environment
FFmpeg API environment
FFmpeg compile
vs2019 installation (win platform)

1.5 Common tools for audio and video development

MediaInfo, analyze video files
VLC player, play test
EasyICE, analyze TS stream
flvAnalyser, analyze FLV
mp4box, analyze mp4
audacity, analyze audio PCM
Elecard_streamEye, analyze H264
Haikang YUVPlayer, analyze YUV

2. The cornerstone of FFmpeg6.0

2.1 FFmpeg commands

Audio PCM/AAC file extraction
Video YUV/H264 file extraction
Demultiplex, multiplex
audio and video recording
Video cropping and merging
Image/Video Conversion
Live streaming push and pull
Watermark/PIP/Nine-square filter

Note: The purpose of mastering FFmpeg: 1. Quickly grasp what FFmpeg can do; 2. Deepen the understanding of audio and video.

2.2 SDL cross-platform multimedia development library practice

SDL environment construction
SDL event handling
SDL threading
Video YUV screen rendering
Audio PCM sound output

Note: SDL is compatible with the three platforms of Win, Ubuntu and Mac, and is mainly used for screen display and sound output of subsequent projects

2.3 The cornerstone of FFmpeg

FFmpeg framework
FFmpeg memory reference counting model
Demultiplexing related AVFormat XXX, etc.
Codec related AVCodec XXX, etc.
Compressed data AVPacket
Uncompressed data AVFrame
FFmpeg object-oriented thinking
Packet/Frame data zero copy

Note: The purpose is to familiarize yourself with the common structure and function interface of FFmpeg

2.4 FFmpeg audio and video demultiplexing + decoding

Demultiplexing process
Audio decoding process
Video decoding process
FLV package format analysis
MP4 Encapsulation Format Analysis
What is the difference between FLV and MP4 seek
Why FLV format can be used for live broadcast
Why MP4 can't be used for live streaming
Can MP4 be used for VOD
AAC ADTS analysis
H264 NALU analysis
AVIO memory input mode
Audio resampling in action
Whether the playback duration of the resampled data is consistent
How to represent PTS after resampling
YUV memory alignment problem after video decoding
PCM arrangement format problem after audio decoding
Hardware decoding dxva2/nvdec/cuvid/qsv
Hardware gpu data transfer to cpu
H265 decoding

Note: FFmpeg API learning: video demultiplexing -> decoding -> encoding -> multiplexing composite video

2.5 FFmpeg audio and video encoding + multiplexing synthetic video

AAC audio encoding
H264 video encoding
PCM+YUV multiplexing MP4/FLV
H264 encoding principle
The difference between IDR frame and I frame
Dynamically modify the encoding bit rate
GOP interval reference value
Multiplex synthesis MP4 audio and video out of sync problem
Encoding and multiplexing timebase problems
MP4 synthesis IOS can not play the problem
How to represent PTS after resampling
Video encoding YUV memory alignment problem
Hardware encoding dxva2/nvenc/cuvid/qsv
H265 encoding principle
H264, H265 code conversion

3. Advanced FFmpeg6.0

3.1 FFmpeg filters

FFmpeg filter chain framework
Audio Filter Framework
Video Filter Framework
multi-channel audio mixing amix
Video watermark watermark
Video area cropping and flipping
video add logo

Note: filter is widely used in the field of video editing.

3.2 ffplay player

Master the meaning of ffplay.c
ffplay framework analysis
demultiplexing thread
audio decoding thread
video decoding thread
sound output callback
Screen Rendering Interval
audio resampling
Screen size format conversion
Differences between audio, video, and external clock synchronization
Audio resampling compensation when referencing video
The essence of volume mute and size adjustment
Audio and video packet queue size limit
Audio and video packet queue thread safety
Audio and video frame queue size limit
Audio and video frame queue thread safety
Pause, play implementation mechanism
Screen stuck problem caused by seek playback
seek playback data queue, synchronous clock processing
How to play frame by frame
Key points of the process of player exit

Note: ffplay.c is the source code of the ffplay command, mastering ffplay will help us develop our own player with twice the result with half the effort.

4. Actual combat of FFmpeg6.0 streaming media project

4.1 FFmpeg+QT player

1. Analysis of open source players

Self-developed framework analysis
Player module division
demultiplexing module
audio and video decoding
player control
Audio and video synchronization
Software decoding + hardware decoding
volume control
Variable speed playback
Drag to jump to play
Screen brightness and saturation adjustment
Screen 4:3 16:9 switching
Code stream information analysis
Screen rotation, flip
Support screenshot

Project Significance: Master the development of the player, including core mechanisms such as player framework analysis, module design, queue design, audio and video synchronization, speed change, and drag and play.

4.2 FFmpeg RTMP recording + streaming project

Introduction to RTMP protocol
Recording + streaming framework analysis
Microphone, system sound capture
Sound Capture Latency Analysis and Resolution
Webcam, screen capture
Screen capture delay analysis and resolution
audio resampling
video scale
AAC encoder package
H264 encoder package
Audio encoding parameter optimization
Video encoding quality/CPU/bit rate/delay optimization
How to set the time stamp when collecting audio and video
Acquisition-encoding-push stream timestamp conversion
Save audio and video as MP4 files
Capture-encode-save mp4 timestamp conversion
Support streaming screen preview
How to support H265 streaming
How to support microphone and system sound mixing
Realize audio acquisition and noise reduction
Will RTMP streaming cause delays?
How to dynamically adjust the code rate of RTMP streaming
How to dynamically adjust the frame rate of RTMP streaming
Can the web side pull RTMP streaming

Project Significance: The streaming module of this project adopts FFmpeg API, and live streaming can still be achieved without a deep understanding of the RTMP protocol. Master the streaming framework and video recording framework, including core technologies such as audio and video collection, module division, audio and video synchronization, encoding optimization, and streaming delay.

4.3 FFmpeg RTMP streaming + video project

Introduction to RTMP protocol
Stream + video frame analysis
Pull flow module package
AAC decoding module package
H264 decoding module package
Packet Queue Design
frame queue design
Realize audio and video synchronization
Is it possible not to do audio and video synchronization analysis
Can play without MetaData
Does RTMP streaming cause delays?
How to detect RTMP streaming delay
How to solve RTMP playback delay
Add support for variable speed playback
Variable speed playback strategy analysis and setting
Analysis and Realization of Second Opening Function Principle
Can ffplay and vlc be used to test playback delay
Realize streaming and save video as MP4
Pull stream, save MP4 timestamp conversion

Note: Master RTMP live streaming + recording framework, including core technologies such as streaming, queue design, audio and video synchronization, low latency, and saving as MP4 files.

4.4 RTSP streaming + recording project

Introduction to RTSP protocol
Realize RTSP streaming + recording based on RTMP streaming project

Project Significance: The streaming module of this project adopts FFmpeg API, and live streaming can still be achieved without a deep understanding of the RTSP protocol. Master the streaming framework and video recording framework, including core technologies such as audio and video collection, module division, audio and video synchronization, encoding optimization, and streaming delay.

4.5 RTSP streaming + video project

Introduction to RTSP protocol
Realize RTSP streaming + video based on RTMP streaming project

Note: Master RTSP streaming + recording framework, including core technologies such as streaming, queue design, audio and video synchronization, low latency, and saving as MP4 files.

5. Project practice

5.1 Self-developed projects and source code analysis

SDL audio player
SDL video playback
FFmpeg memory reference count
MP4/FLV/TS demultiplexing
Extract H264
Extract AAC
H264/H265 decoding
AAC/MP3 decoding
AVIO memory input mode
audio resampling
Video size format conversion
DXVA2 decoding
NVDEC decoding
CUVID decoding
QSV decoding
AAC encoding optimization
H264 encoding optimization
H265 encoding optimization
AAC to MP3
H264 to H265
DXVA2 encoding
NVENC encoding
CUVID encoding
QSV encoding
MP4 synthesis
FLV synthesis
Multiplex audio mixing
add watermark to video
Video area cropping
video flip
video add logo
ffplay player
Self-developed QT player
RTMP streaming
RTMP streaming
RTSP streaming
RTSP streaming
HLS streaming

6. Learning method

Read the document first, then watch the video
Run and debug the course code first, then modify and debug, and finally try to rewrite it yourself
The knowledge points of each lesson are summarized into a technical blog (there is a blog contest every month, with bonus support)
Combining courses and work to rethink demand functions
Communicate with the teacher about the technical points that you don’t understand
Partial audio and video client direction: server-related content is mainly for understanding, such as SRS streaming media server, mainly for understanding the push-pull streaming framework and protocol conversion logic, and does not need to spend too much time reading the source code.
Streaming media server direction: The design of the UI interface is based on understanding. For example, the player project only needs to focus on audio and video related content.

[Learning address]: Advanced development of FFmpeg/WebRTC/RTMP/NDK/Android audio and video streaming media
[Article Benefits]: Receive more audio and video learning materials packages, Dachang interview questions, technical videos and learning roadmaps for free, including ( C/C++, Linux, FFmpeg webRTC rtmp hls rtsp ffplay srs, etc.) If you need it, you can click 1079654574 to join the group to receive it~

FFmpeg learning books

"FFmpeg From Getting Started to Mastering " is a practical technical guide introducing FFmpeg. The book has 10 chapters and is divided into two parts. Part (chapters 1 to 7) is the command line use of FFmpeg, which introduces the basic components of FFmpeg, the use of FFmpeg tools, the packaging operation of FFmpeg, the transcoding operation of FFmpeg, the streaming media operation of FFmpeg, and the filter operation of FFmpeg , FFmpeg equipment operation.

The second part (Chapter 8~10) is the API usage of FFmpeg, which introduces the API usage operation of the FFmpeg encapsulation part, the API usage operation of the FFmpeg codec part, and the API usage operation of the FFmpeg filter part. The method is described, including the operation of the new API and the old API.

This book is suitable for engineers engaged in multimedia server-side development and embedded video processing engineers. This book is also suitable as a computer professional undergraduate and graduate students, especially the direction of multimedia research as a reference book.

There is also a pdf version of <<ffmpeg Source Code Analysis>> circulated on the Internet, and I haven’t seen other books about ffmpeg so far; here is one more thing, different versions of ffmpeg may have different apis. For details, everyone You can go to the ffmpeg official website to view the manual instructions!