This article is respectively :
- Introduction to FFmpeg
- FFmpeg audio and video decoding playback
- Common problems of Clang compiling FFmpeg
Let’s talk about today. Let’s talk about FFmpeg first today.
1. Introduction to FFmpeg
1 Introduction
FFmpeg (FastForward Mpeg) is an open source software that follows the GPL. It performs very well in audio and video processing, covering almost all existing video and audio formats for encoding, decoding, transcoding, mixing, filtering and playback. It is also a cross-platform software, perfectly compatible with Linux, Windows, Mac OSX and other platforms. In fact, it consists of 3 major components:
FFmpeg: Composed of command lines, used for multimedia format conversion FFplay: A multimedia player based on the FFmpeg open source code library libraries FFprobe: A multimedia stream analyzer based on FFmpeg
2. Two powerful functions of FFmpeg
1. Command function 2. API function
2.1 Command function
How to use the app
ffmpeg –y –i input –vcodeccopy –an output.aviAmong them, -y means to overwrite the file with the same name, -i means the input file is bus.avi, -vcodec means the encoding method, the following copy means to use the original encoding mode, that is, no re-encoding, -an means to remove the audio, and the following busv.avi means the separated video file.
Similarly, the command behavior of separating the audio file from the video is as follows:
ffmpeg -ibus.avi -acodec copy -vn busa.wavThe above examples illustrate the usage of the application. The command line of the application is much simpler than the code, and can also realize various functions such as audio and video separation, transcoding, and playback. For example, the command line behavior of video transcoding:
ffmpeg -y -i input.mp4 -vcodec libx264 -acodec copy output.mp4This command is used to cut the video. -ss indicates the start from the second. In the above example, it starts from the 5th second. -t indicates to cut the video that lasts for several seconds. In the above example, it is to cut the video with a length of 10 seconds. copy indicates that the video encoding format and audio encoding format are consistent with the original video.
ffmpeg -ss 0:0:5 -t 0:0:10 -i input.avi -vcodec copy -acodec copy output.aviSeparate video and audio streams
ffmpeg -i input_file -vcodec copy -an output_file_video //分离视频流
ffmpeg -i input_file -acodec copy -vn output_file_audio //分离音频流video demultiplexing
ffmpeg –i test.mp4 –vcodec copy –an –f m4v test.264
ffmpeg –i test.avi –vcodec copy –an –f m4v test.264video transcoding
ffmpeg –i test.mp4 –vcodec h264 –s 352*278 –an –f m4v test.264 //转码为码流原始文件
ffmpeg –i test.mp4 –vcodec h264 –bf 0 –g 25 –s 352*278 –an –f m4v test.264 //转码为码流原始文件
ffmpeg –i test.avi -vcodec mpeg4 –vtag xvid –qsame test_xvid.avi //转码为封装文件video encapsulation
ffmpeg –i video_file –i audio_file –vcodec copy –acodec copy output_filevideo cut
ffmpeg –i test.avi –r 1 –f image2 image-%3d.jpeg //提取图片
ffmpeg -ss 0:1:30 -t 0:0:20 -i input.avi -vcodec copy -acodec copy output.avi
//剪切视频
//-r 提取图像的频率，-ss 开始时间，-t 持续时间video recording
ffmpeg –i rtsp://192.168.3.205:5555/test –vcodec copy out.aviYUV sequence playback
ffplay -f rawvideo -video_size 1920x1080 input.yuvYUV sequence to AVI
ffmpeg –s w*h –pix_fmt yuv420p –i input.yuv –vcodec mpeg4 output.aviDescription of common parameters:
The main parameters:
- -i set the input stream
- -f set the output format
- -ss start time video parameters:
- -b Set the video traffic, the default is 200Kbit/s
- -r set the frame rate, the default is 25
- -s sets the width and height of the screen -aspect sets the ratio of the screen
- -vn do not process video
- -vcodec sets the video codec, if not set uses the same codec as the input stream audio parameters:
- -ar sets the sampling rate -ac sets the number of channels of the sound
- -acodec set the audio codec, if not set use the same codec as the input stream -an do not process audio
2. FFmpeg audio and video decoding and playback
foreword
Usually, media files are stored in our computers, mobile phones and other devices in formats such as MP4, MKV, FLV, etc., and these file formats are all packaged formats, that is, audio and video data are packaged into files according to corresponding specifications.
1. FFmpeg audio and video decoding process
Usually when we play media files, we usually need to go through the following steps
2. FFmpeg audio and video decoding principle
2.1. Solution agreement
Parse the data of the streaming media protocol into the corresponding standard encapsulation format data. When video and audio are transmitted on the network, various streaming media protocols are often used, such as HTTP, RTMP, or MMS and so on. While transmitting video and audio data, these protocols also transmit some signaling data. These signaling data include control of playback (play, pause, stop), or description of network status.
In the process of decomposing the protocol, the signaling data will be removed and only the video and audio data will be kept. For example, the data transmitted by RTMP protocol will output the data in FLV format after the protocol solution operation.
2.2. Decapsulation
The input data in the encapsulation format is separated into audio stream compression coded data and video stream compression coded data. There are many types of encapsulation formats, such as MP4, MKV, RMVB, TS, FLV, AVI, etc. Its function is to put the compressed and encoded video data and audio data together in a certain format. For example, after decapsulating the data in FLV format, the H.264-encoded video stream and AAC-encoded audio stream are output.
2.3. Decoding
The video/audio compression coded data is decoded into uncompressed video/audio raw data. Decoding is the most important and complex part of the whole system. Through decoding, the compressed and encoded video data output becomes uncompressed color data, such as YUV420P, RGB, etc.;
2.4. Audio and video synchronization
According to the parameter information obtained during the processing of the decapsulation module, the decoded video and audio data are synchronized, and the video and audio data are sent to the graphics card and sound card of the system for playback.
2.5.FFmpeg audio and video decoding
From the above, we know that each media file mainly goes through two key steps before being played by the terminal, which are decapsulation and decoding. In ffmpeg, the decapsulation and decoding process is implemented using related interfaces as shown in the figure below:
As can be seen from the above figure, we need to focus on the following API interfaces of FFmpeg:
- av_register_all(): Register all components.
- avformat_open_input(): Open the input video file.
- avformat_find_stream_info(): Get video file information.
- avcodec_find_decoder(): Find the decoder.
- avcodec_open2(): Open the decoder.
- av_read_frame(): Read a frame of compressed data from the input file.
- avcodec_decode_video2(): Decode a frame of compressed data.
3. Use of FFmpeg interface
Before using FFmpeg to decode media files, you first need to register the container and codec-related components.
av_register_all()If we need to play network multimedia, we can load socket libraries and libraries related to network encryption protocols to provide support for subsequent network use.
avformat_network_init();We use avformat_open_input()to open a media file and get the context of the media file packaging format
//打开一个文件并解析。可解析的内容包括：视频流、音频流、视频流参数、音频流参数、视频帧索引
int res = avformat_open_input(&pAVFormatCtx, url, NULL, NULL);
LOGI("avformat_open_input %s %d", url, res);
if(res != 0){
LOGE("can not open url :%s", url);
callJava->onCallError(CHILD_THREAD, 1001, "can not open url");
exit = true;
pthread_mutex_unlock(&init_mutex);
return;
}By avformat_find_stream_info()obtaining the media file, extracting the context information of the stream, and separating the audio and video streams.
//解码时，作用是从文件中提取流信，将所有的Stream的MetaData信息填充好，先read_packet一段数据解码分析流数据
if(avformat_find_stream_info(pAVFormatCtx, NULL) < 0){
LOGE("can not find streams from %s", url);
callJava->onCallError(CHILD_THREAD, 1002,"can not find streams from url");
exit = true;
pthread_mutex_unlock(&init_mutex);
return;
}Find the audio stream or video stream in the file by traversing
for(int i = 0; i < pAVFormatCtx->nb_streams; i++){
if(pAVFormatCtx->streams[i]->codecpar->codec_type == AVMEDIA_TYPE_AUDIO) {
//得到音频流
if(audio == NULL){
audio = new FFAudio(playstatus, pAVFormatCtx->streams[i]->codecpar->sample_rate, callJava);
audio->streamIndex = i;
audio->codecpar = pAVFormatCtx->streams[i]->codecpar;
audio->duration = pAVFormatCtx->duration / AV_TIME_BASE;
audio->time_base = pAVFormatCtx->streams[i]->time_base;
duration = audio->duration;
//av_q2d(time_base)=每个刻度是多少秒
LOGI("audio stream_info[%d], duration:%d, time_base den:%d,sample_rate:%d",i, audio->duration, audio->time_base.den, pAVFormatCtx->streams[i]->codecpar->sample_rate);
LOGI("audio stream_info[%d], duration %lld", i, pAVFormatCtx->duration);
}
} else if (pAVFormatCtx->streams[i]->codecpar->codec_type ==AVMEDIA_TYPE_VIDEO){
//得到视频流
if (video == NULL){
video = new FFVideo(playstatus, callJava);
video->streamIndex = i;
video->codecpar = pAVFormatCtx->streams[i]->codecpar;
video->time_base = pAVFormatCtx>streams[i]->time_base;
int num = pAVFormatCtx->streams[i]->avg_frame_rate.num;
int den = pAVFormatCtx->streams[i]->avg_frame_rate.den;
LOGI("video stream_info[%d], frame_rate num %d,den %d", i, num,den);
if(num != 0 && den != 0){
int fps = num / den;//[25 / 1]
video->defaultDelayTime = 1.0 / fps;
}
LOGI("video stream_info[%d], defaultDelayTime is %f", i, video->defaultDelayTime);
}
}
}After separating the audio and video streams, you can find the corresponding AVCodecContext, which is the context of the codec, to find the corresponding decoder and set it.
//查找对应的解码器 存储编解码器信息的结构体
AVCodec *avCodec = avcodec_find_decoder(codecpar->codec_id);// 软解
//avCodec = avcodec_find_decoder_by_name("mp3_mediacodec"); // 硬解
if (!avCodec){
LOGE("MFFmpeg::getCodecContext can not find decoder!");
callJava->onCallError(CHILD_THREAD, 1003, "can not find decoder");
exit = true;
pthread_mutex_unlock(&init_mutex);
return -1;
}
LOGI("getCodecContext codecpar-> 解码类型:%d 编码格式:%s" , codecpar->codec_type, avCodec->name);
//配置解码器
*avCodecContext = avcodec_alloc_context3(avCodec);
if (!*avCodecContext){
LOGE("can not alloc new decodecctx");
callJava->onCallError(CHILD_THREAD, 1004, "can not alloc new decodecctx");
exit = true;
pthread_mutex_unlock(&init_mutex);
return -1;
}Decode the media file by avcodec_open2()opening the codec.
//打开编解码器
if(avcodec_open2(*avCodecContext, avCodec, 0) != 0){
LOGE("cant not open strames");
callJava->onCallError(CHILD_THREAD, 1006, "cant not open strames");
exit = true;
pthread_mutex_unlock(&init_mutex);
return -1;
}So the relationship used in the four steps 2, 3, 4, and 5 is as follows
After opening the decoder, av_read_frame()the compressed data is read frame by frame.
AVPacket \*avPacket = av\_packet\_alloc();
//读取具体的音/视频帧数据
int ret = av_read_frame(pAVFormatCtx, avPacket);
if (ret==0){
//stream_index：标识该AVPacket所属的视频/音频流
if(avPacket->stream_index == audio->streamIndex){
//LOGI("audio 解码第 %d 帧 DTS:%lld PTS:%lld", count, avPacket->dts,avPacket->pts);
audio->queue->putAVpacket(avPacket);
} else if(avPacket->stream_index == video->streamIndex){
//LOGI("video 解码第 %d 帧 DTS:%lld PTS:%lld", count, avPacket->dts,avPacket->pts);
count++;
video->queue->putAVpacket(avPacket);
} else{
av_packet_free(&avPacket);
av_free(avPacket);
avPacket = NULL;
}
}By avcodec_decode_video2()/avcodec_decode_audio4decoding a frame of video or audio compressed data, the video pixel data is obtained through AVPacket->AVFrame.
//解码AVPacket->AVFrame
ret = avcodec_decode_audio4(pCodeCtx, frame, &got_frame, packet);
//解码一帧视频压缩数据，得到视频像素数据
ret = avcodec_decode_video2(pCodecCtx, pFrame, &got_picture, packet);3. Common problems of Clang compiling FFmpeg
1. Command not found
Error message:
./build_android.sh: line 18: --enable-shared: command not found
./build_android.sh: line 20: --disable-static: command not found
./build_android.sh: line 22: --disable-doc: command not found
./build_android.sh: line 24: --disable-ffmpeg: command not found
./build_android.sh: line 26: --disable-ffplay: command not found
./build_android.sh: line 28: --disable-ffprobe: command not found
./build_android.sh: line 30: --disable-ffserver: command not found
./build_android.sh: line 32: --disable-avdevice: command not foundSolution: If you directly copy the shell script on the Internet, it may be in dos format. Please use dos2unix build_android.sh to convert it and delete the extra spaces (this is very important. dos2unix is a tool. If you have not installed it, please install it first: brew install dos2unix, it will be over soon.
2.xmakefile is not generated
Error message:
./android_config.sh: line 36: --enable-shared: command not found
Makefile:2: ffbuild/config.mak: No such file or directory
Makefile:40: /tools/Makefile: No such file or directory
Makefile:41: /ffbuild/common.mak: No such file or directory
Makefile:91: /libavutil/Makefile: No such file or directory
Makefile:91: /ffbuild/library.mak: No such file or directory
Makefile:93: /fftools/Makefile: No such file or directory
Makefile:94: /doc/Makefile: No such file or directory
Makefile:95: /doc/examples/Makefile: No such file or directory
Makefile:160: /tests/Makefile: No such file or directory
make: *** No rule to make target `/tests/Makefile'. Stop.
Makefile:2: ffbuild/config.mak: No such file or directory
Makefile:40: /tools/Makefile: No such file or directory
Makefile:41: /ffbuild/common.mak: No such file or directory
Makefile:91: /libavutil/Makefile: No such file or directory
Makefile:91: /ffbuild/library.mak: No such file or directory
Makefile:93: /fftools/Makefile: No such file or directory
Makefile:94: /doc/Makefile: No such file or directory
Makefile:95: /doc/examples/Makefile: No such file or directory
Makefile:160: /tests/Makefile: No such file or directorySolution: Execute ./configure --disable-x86asm to generate config.mak file
3.arm-linxu-androideabi-gcc is unable to
create an executable file
Error message:
/Users/aria/dev/android/sdk/ndk-bundle/toolchains/arm-linux-androideabi-
4.9/prebuilt/darwin-x86_64/bin/arm-linux-androideabi-gcc is unable to create an
executable file.Reason : Check the ndk version, android has officially removed the gcc compilation tool starting from r18b. See ndk r18b revision content for details. Solution : Use clang to compile
4./android_config.sh: line 32: xxxxx No such file or directory
Reason : The commands after .configure cannot have comments
Solution: Which line of code to delete the comment
5、static declaration of 'xxx' follows non-static declaration
Solution : config.h searches for characters such as lrint, lrintf, round, roundf, etc., and changes 0 to 1
#define HAVE_LLRINT 1
#define HAVE_LLRINTF 1
#define HAVE_LRINT 1
#define HAVE_LRINTF 1
#define HAVE_ROUND 1
#define HAVE_ROUNDF 1
#define HAVE_CBRT 1
#define HAVE_CBRTF 1
#define HAVE_COPYSIGN 1
#define HAVE_TRUNC 1
#define HAVE_TRUNCF 1
#define HAVE_RINT 1
#define HAVE_HYPOT 1
#define HAVE_ERF 1Or use sed directly to modify the config.h file
sed -i -e 's/#define HAVE_LLRINT 0/#define HAVE_LLRINT 1/g' config.h
sed -i -e 's/#define HAVE_LLRINTF 0/#define HAVE_LLRINTF 1/g' config.h
sed -i -e 's/#define HAVE_LRINT 0/#define HAVE_LRINT 1/g' config.h
sed -i -e 's/#define HAVE_LRINTF 0/#define HAVE_LRINTF 1/g' config.h
sed -i -e 's/#define HAVE_ROUND 0/#define HAVE_ROUND 1/g' config.h
sed -i -e 's/#define HAVE_ROUNDF 0/#define HAVE_ROUNDF 1/g' config.h
sed -i -e 's/#define HAVE_CBRT 0/#define HAVE_CBRT 1/g' config.h
sed -i -e 's/#define HAVE_CBRTF 0/#define HAVE_CBRTF 1/g' config.h
sed -i -e 's/#define HAVE_COPYSIGN 0/#define HAVE_COPYSIGN 1/g' config.h
sed -i -e 's/#define HAVE_TRUNC 0/#define HAVE_TRUNC 1/g' config.h
sed -i -e 's/#define HAVE_TRUNCF 0/#define HAVE_TRUNCF 1/g' config.h
sed -i -e 's/#define HAVE_RINT 0/#define HAVE_RINT 1/g' config.h
sed -i -e 's/#define HAVE_HYPOT 0/#define HAVE_HYPOT 1/g' config.h
sed -i -e 's/#define HAVE_ERF 0/#define HAVE_ERF 1/g' config.h
sed -i -e 's/#define HAVE_GMTIME_R 0/#define HAVE_GMTIME_R 1/g' config.h
sed -i -e 's/#define HAVE_LOCALTIME_R 0/#define HAVE_LOCALTIME_R 1/g' config.h
sed -i -e 's/#define HAVE_INET_ATON 0/#define HAVE_INET_ATON 1/g' config.h6、xxxxxxxxxx error: expected ')'
Error message:
#define getenv(x) NULL
^
/home/cd008/diska/android-ndk-r9/platforms/android-18/archarm/usr/include/stdlib.h:54:14: note: in expansion of macro 'getenv'
extern char *getenv(const char *);
^
./config.h:17:19: error: expected ')' before numeric constant
#define getenv(x) NULL
^
/home/cd008/diska/android-ndk-r9/platforms/android-18/archarm/usr/include/stdlib.h:54:14: note: in expansion of macro 'getenv'
extern char *getenv(const char *);Solution: Comment out #define getenv(x) NULL /#define getenv(x) NULL/ in config.h
sed -i -e 's/#define getenv(x) NULL/\/\*#define getenv(x) NULL\*\//g' config.h7、arm-linux-androideabi-ld -Wl,- soname,libavutil.so unknown option
Error message:
Users/aria/dev/android/sdk/ndk-bundle/toolchains/arm-linux-androideabi-
4.9/prebuilt/darwin-x86_64/bin/arm-linux-androideabi-ld -Wl,-sonameReason : The command for gcc to build .so is -shared -wl,soname,xxxx.so and clang is -shared -soname xxx.so
Solution : Modify ffbuild/config.mak the file and SHFLAGS=-shared -Wl,-soname,$(SLIBNAME) change it to SHFLAGS=- shared -soname $(SLIBNAME)
If you are interested in audio and video development and think the article is helpful to you, don't forget to like and bookmark it! Or have your own views on some of the explanations in this article. If you have any questions, please discuss them in the comment area below!
Original address: Android FFmpeg audio and video decoding and playback
★The business card at the end of the article can receive audio and video development learning materials for free, including (FFmpeg, webRTC, rtmp, hls, rtsp, ffplay, srs) and audio and video learning roadmaps, etc.
see below!