Use FFmpeg to play audio and video processing (operations)

I. Introduction

To play audio and video transcoding, streaming, etc., you must first understand what platform you are using. Different platforms have different ways to obtain cameras and microphones

1.1 View device information

1.1.1 Windows platform

 Right click on "Computer" - "Management" - "Device Manager" to view the information of the camera and microphone

Or query related information through ffmpeg:

ffmpeg -list_devices true -f dshow -i dummy

Note: Windows platform can use dshow to display all devices, and Linux platform needs to display devices by type, so you need to pay attention to the differences in parameters such as dshow, alsa, and v4l2 in the command:

dshow: Refers to DirectShow, which is a streaming media framework for the Windows platform, providing high-quality multimedia stream collection and return visit functions
alsa: refers to Advanced Linux Sound Architecture, which is an advanced Linux sound architecture that provides audio and MIDI support
Video4linux2: V4L2 for short , is the kernel driver for video devices on the Linux platform, providing APIs required for image, video, and audio collection

Subsequent commands in this article need to replace relevant parameters according to your own platform

1.1.2 Linux platform

1) Microphone Management

There are many ways to query the microphone on Linux, but pay attention to whether it can be used for ffmpeg

a. Query microphone information

# 列出所有声卡设备
aplay -l	

# 列出所有录音设备
arecord -l	

# 列出所有录音设备（获取音频采集时声卡名称用这个）
arecord -L	

# 查看card2信息
ll /proc/asound/card2	

# 查看声卡支持的格式（通道信息、支持的采样率等）
cat /proc/asound/card2/stream0	

 b. Configure the sound card

# 安装pavucontrol控制器软件
sudo apt install pavucontrol

# 使用pavucontrol控制器配置声卡
pavucontrol

c. Use the sound card to record

arecord -D "plughw:1,0" -f S16_LE -r 44100 -d 5 -t wav file.wav

# 参数说明：
# -D	设备名称，“plughw:1,0”中1表示card1，0表示subdevice
# -r	采样率
# -f	录音格式，S16_LE表示16bit位宽
# -d	录音时长
# -t	输出音频格式

d. Play the recording

aplay file.wav

2) Camera management

Query camera information

ls /dev/video*

Or install the v4l2 tool to view

# 安装v4l2工具包
sudo apt install v4l-utils

# 查看摄像头设备
sudo v4l2-ctl --list-devices

# 查看当前摄像头支持的视频压缩格式
$ sudo v4l2-ctl -d /dev/video0 --list-formats

# 查看摄像头所有参数
$ sudo v4l2-ctl -d  /dev/video0 --all

 # 查看摄像头所支持的分辨率
 sudo v4l2-ctl --list-framesizes=MJPG -d /dev/video0

 

2. FFmpeg package

        FFmpeg may be the most powerful audio and video codec tool free software, it is also an audio and video editing suite. It can implement functions such as audio and video encoding, decoding, transcoding, stream processing, audio and video filtering, and audio and video playback. It is also highly portable and can be built in a variety of environments, across platforms such as Linux, Mac OS, Windows, BSD, Solaris, and more. It contains libacvodec, libavutil, libavformat, libavfilter, libavdevice, libswscale, and libswresample components that can be used by applications, while ffmpeg includes tools such as ffmpeg, ffplay, ffprobe, etc. Users can use these tools and components to achieve free transcoding and playback of audio and video.

2.1 Using FFmpeg

2.1.1 ffmpeg parameter description

Public parameters:
    -i specifies the input source
    -codec refers to direct copy stream
    -vn removes video (does not process video)
    -an removes audio (does not process audio)
    -t duration
    -ss start time
    -f output format or encoding (h264 means Video encoding during output; mp4 indicates that the output is in mp4 video format; mpegts indicates that the output is ts video stream; rtsp indicates that the output is rtsp video stream)

Video parameters:
    -vframes set the number of video frames to be output
    -b:v video bit rate
    -r frame rate (default 25)
    -s frame size (W*H)
    -aspect aspect ratio (4:3/16:9)
    - vcodec video option, generally followed by copy to indicate that the original decoded data must be copied
    -vf video filter

Audio parameters:
    -aframes set the number of audio frames to be output
    -b:a audio code rate
    -ar audio sampling rate
    -ac set the number of sound channels (usually left and right channels, that is, 2)
    -acodec audio options, generally followed by copy indicates that the original decoded data must be copied
    -af audio filter

 2.1.2 Local audio and video processing

1) Video format conversion

# 视频格式转换（将h264格式转为ts格式）
ffmpeg -i test.h264 -vcodec copy -f mpegts test.ts
# 视频格式转换（将h264格式转为MP4格式）
ffmpeg -i test.h264 -vcodec copy -f mp4 test.mp4

# RGB和YUV数据转换
ffmpeg -s 640*480 -i yuv420p_640x480.yuv  -pix_fmt yuv420p -pix_fmt rgb24 -s 640*480 rgb24_640x480.rgb

2) Video transcoding

# 改变视频编码格式
ffmpeg -i test_1280x720.mp4 -vcodec libx265 -acodec libmp3lame out_h265_mp3.mkv
# 改变帧率
ffmpeg -i test_1280x720.mp4 -r 15 output2.mp4
# 改变视频码率
ffmpeg -i test.mp4 -b 400k output_b.mkv
# 改变视频码率（仅视频）
ffmpeg -i test.mp4 -b:v 400k output_bv.mkv
# 改变视频码率（仅音频）
ffmpeg -i test.mp4 -b:a 192k output_ba.mp4
# 改变视音频码率
ffmpeg -i test.mp4 -b:v 400k -b:a 192k output_bva.mp4
# 改变视频分辨率
ffmpeg -i test.mp4 -s 480x270 output_480x270.mp4
# 视频码率调整
ffmpeg -i test.mp4 -vframes 300 -b:v 400k -r 30 -s 640x480 -aspect 16:9 -vcodec libx265
# 改变音频采样率
ffmpeg -i test.mp4 -ar 44100 output_44100hz.mp4

3) Audio and video extraction

# 直接提取视频画面
ffmpeg -i test_1280x720.mp4 -vcodec copy -an video.mp4
# 提取视频画面并强制格式
ffmpeg -i test_1280x720.mp4 -vcodec libx264 -an video.h264
# 直接提取视频音轨
ffmpeg -i test_1280x720.mp4 -acodec copy -vn audio.mp4
# 提取视频音轨并强制编码
ffmpeg -i test_1280x720.mp4 -acodec libmp3lame -vn audio.mp3
# 提取指定码率的音频
ffmpeg -i test_1280x720.mp4 -b:a 192k -ar 48000 -ac 2 -acodec aac -aframes 200 out2.mp3

# 提取yuv数据（提取3秒，分辨率和源视频保持一致）
ffmpeg -i test_1280x720.mp4 -t 3 -pix_fmt yuv420p yuv420p_orig.yuv
# 提取yuv数据（提取3秒，分辨率转为320*240）
ffmpeg -i test_1280x720.mp4 -t 3 -pix_fmt yuv420p -s  640*480 yuv420p_320x240.yuv

# 提取rgb数据（提取3秒，分辨率转为320*240）
ffmpeg -i test.mp4 -t 3 -pix_fmt rgb24 -s 320x240 rgb24_320x240.rgb

 4) Audio and video cropping

# 视频裁剪（从第10秒到第20秒）
ffmpeg -i input.mp4 -ss 00:00:10 -t 00:00:20 -c:v copy -c:a copy output.mp4
# 视频拆分（每10秒拆分成一个视频文件）
ffmpeg -i input.mp4 -f segment -segment_time 10 -c copy output%d.mp4
# 视频截图（在test.mp4中截图）
ffmpeg -i test.mp4  -t 0.001 -s 352x240 1.jpg
# 视频截图（将test.mp4前30帧做成gif动图）
ffmpeg -i test.mp4 -vframes 30 -y -f gif 1.gif
# 视频截图（从视频前10s中取图像，1s提取一帧）
ffmpeg -i test.mp4 -t 10 -r 1 pic-%03d.jpg

5) Audio and video synthesis

# 查看视频信息（显示音视频轨道信息、码率等）
ffmpeg -i Hotel.Transylvania.4.Transformania.2022.中英字幕.mkv

# 视频中合并字幕，并强制视频编码为H264编码
ffmpeg -y -i「视频全名」 -vf subtitles=「字幕文件名」 -vcodec h264 「导出的视频文件名」
# 视频中合并字幕，并强制视频编码为x264编码（此方式提高crf值，但清晰度很高）
ffmpeg -y -i filename.mkv -vf subtitles='filename.mkv' -disposition:s default+forced -c:v libx264 -c:a libmp3lame -crf 27 -preset ultrafast filename.mp4 

# 视频剥离多音轨和字幕（仅保留视频轨道和音频默认轨道）
ffmpeg -i Hotel.Transylvania.4.Transformania.2022.中英字幕.mkv -map 0:0 -map 0:1  OutPut.mkv

# 截取并融合字幕轨道到视频轨道中（适用于subrip类型字幕）
ffmpeg -i Hotel.Transylvania.4.Transformania.2022.中英字幕.mkv -filter_complex "[0:v:0]subtitles=Hotel.Transylvania.4.Transformania.2022.中英字幕.mkv:si=2[v]" -map "[v]" -map 0:1  Hotel.mp4
## 注：
	-filter_complex "[0:v:0]subtitles=Hotel.Transylvania.4.Transformania.2022.中英字幕.mkv:si=2[v]" -map "[v]" 	表示截取并融合视频文件的视频轨道与字母轨道，融合后再与音频轨道融合并渲染
# 截取并融合字幕轨道到视频轨道中（适用于hdmv_pgs_subtitle类型字幕）
ffmpeg -i HotelTransylvania.mkv -filter_complex "[0:v][0:s]overlay[v]" -map "[v]" -map 0:a:0 out.mp4

# 视频上增加水印（文字水印）
ffmpeg -i input.mp4 -vf "drawtext=text='Your Watermark':fontsize=20:fontcolor=white:x=10:y=10" -c:v libx264 -crf 18 output.mp4
# 视频上增加水印（图片水印）
ffmpeg -i input.mp4 -i watermark.png -filter_complex "overlay=10:10" -c:v libx264 -crf 18 output.mp4

# 视频拼接（将test1.h264和test2.h264视频拼接起来，输出out12.h264视频）
ffmpeg -i "concat:test1.h264|test2.h264" -vcodec copy -f h264 out12.h264
# 视频双宫格显示
ffmpeg -i 1.mp4 -i 2.mp4 -i 3.mp4 -i 4.mp4 -filter_complex "nullsrc=size=640x480[base];[0:v]setpts=PTS-STARTPTS,scale=320x240[upperleft];[1:v]setpts=PTS-STARTPTS,scale=320x240[upperright];[base][upperleft]overlay=shortest=1[tmp1];[tmp1][upperright]overlay=shortest=1:x=320" out2.mp4
# 视频四宫格显示
ffmpeg -i 1.mp4 -i 2.mp4 -i 3.mp4 -i 4.mp4 -filter_complex "nullsrc=size=640x480[base];[0:v] setpts=PTS-STARTPTS,scale=320x240[upperleft];[1:v]setpts=PTS-STARTPTS,scale=320x240[upperright];[2:v]setpts=PTS-STARTPTS, scale=320x240[lowerleft];[3:v]setpts=PTS-STARTPTS,scale=320x240[lowerright];[base][upperleft]overlay=shortest=1[tmp1];[tmp1][upperright]overlay=shortest=1:x=320[tmp2];[tmp2][lowerleft]overlay=shortest=1:y=240[tmp3];[tmp3][lowerright]overlay=shortest=1:x=320:y=240" out3.mp4

 2.1.3 Audio and video stream processing

1) Capture local camera, microphone

# 捕获摄像头视频
ffmpeg -f dshow -i video="Integrated Camera" -vcodec libx264 mycamera.mkv

# 捕获麦克风声音
ffmpeg -f dshow -i audio="麦克风(High Definition Audio 设备)" -acodec aac d:\temp.aac

# 捕获桌面和声卡声音录制成flv文件
ffmpeg -f dshow -i video="screen-capture-recorder" -f dshow -i audio="virtual-audio-capturer" -pix_fmt yuv420p -ar 48000 -vcodec libx264 -crf 23 -preset veryslow -x264opts b-adapt=2:bframes=0:aq-strength=1:psy-rd=0.8,0 -vsync vfr -acodec aac -bsf:a aac_adtstoasc -f flv temp.flv
## 注：
##	上面使用x264编码，若想提高x264编码速度，可使用-preset:v ultrafast -tune:v zerolatency  两个参数
##	如：ffmpeg -f dshow -i video="Integrated Webcam" -vcodec libx264 -preset:v ultrafast -tune:v zerolatency d:\test.mp4

# 创建空白视频（指定颜色与时长）
ffmpeg -ss 0 -t 00:00:05.300  -f lavfi -i color=c=0x000000:s=1280x720:r=25 -vcodec libx264 D:/linux-share-dir/video_file/video/test.mp4
## 注：
##	-t 是视频的时长
##	color=c=0x000000:s=1280x720:r=25 画面颜色、尺寸、帧率

 2) Local camera and microphone streaming

# 查看本机设备信息
ffmpeg -list_devices true -f dshow -i dummy

## 注：本机硬件推送流发出后，必须有配套的RTMP/RTSP服务器接收数据流

# 推送音视频流到RTMP服务器（Windows平台）
ffmpeg -f dshow -i video="Lenovo EasyCamera":audio="麦克风 (Realtek High Definition Audio)" -vcodec libx264 -acodec copy -preset:v ultrafast -tune:v zerolatency -f flv "rtmp://192.168.1.85:8553/test"

# 推送音视频流到RTSP服务器（Windows平台）
ffmpeg -f dshow -i video="Lenovo EasyCamera":audio="麦克风 (Realtek High Definition Audio)" -vcodec libx264 -acodec libvo_aacenc -preset:v ultrafast -tune:v zerolatency -f rtsp rtmp://192.168.1.85:8554/test
# 推送音视频流到RTSP服务器（设置帧率）
ffmpeg -f dshow -i video="Lenovo EasyCamera":audio="麦克风 (Realtek High Definition Audio)" -vcodec libx264 -acodec libvo_aacenc -b 1080k -r 25 -preset:v ultrafast -tune:v zerolatency -f rtmp://192.168.1.85:8554/test
## 注：设置帧率信息要在推流地址前面，在摄像头信息后面

# 推送音频流到RTSP服务器（Linux默认声卡）
ffmpeg -f alsa -i default -ac 2 -ar 44100 -acodec aac -f rtsp rtsp://192.168.1.204:8554/audio
# 推送音频流到RTSP服务器（Linux指定声卡，查询声卡信息可以使用arecord -L获取声卡名称）
ffmpeg -f alsa -i plughw:CARD=sbj3308,DEV=0 -ac 2 -ar 44100 -acodec aac -f rtsp rtsp://192.168.1.204:8554/audio

# RTMP/RTSP方式推麦克风
ffmpeg -f alsa -thread_queue_size 1024 -ac 2-ar 44100 -i hw:0,0 -acodec aac -f flv rtmp://192.168.8.222:1935/live/stream0
ffmpeg -f alsa -thread_queue_size 1024 -ac 2-ar 44100 -i hw:0,0 -acodec aac -f rtsp rtsp://192.168.8.222:1935/live/stream0

# RTMP/RTSP方式推摄像头
ffmpeg -f video4linux2 -r 30 -i /dev/video0 -vcodec h264 -f flv rtmp://192.168.8.222:1935/live/stream0
ffmpeg -f video4linux2 -r 30 -i /dev/video0 -vcodec h264 -f rtsp rtsp://192.168.8.222:1935/live/stream0

# RTMP/RTSP方式推摄像头和麦克风
ffmpeg -f video4linux2 -r 30 -i /dev/video0 -vcodec h264 -f alsa -thread_queue_size 1024 -ac 2 -ar 44100 -i hw:0,0 -acodec mp3 -f flv rtmp://192.168.8.222:1935/live/stream0
ffmpeg -f video4linux2 -r 30 -i /dev/video0 -vcodec h264 -f alsa -thread_queue_size 1024 -ac 2 -ar 44100 -i hw:0,0 -acodec mp3 -f rtsp rtsp://192.168.8.222:1935/live/stream0

# 生成空的视频结合麦克风音频输出RTSP流
ffmpeg -f lavfi -i color=c=0x000000:s=1280*720:r=25 -vcodec libx264 -f alsa -i plughw:CARD=sbj3308,DEV=0 -ac 2 -ar 44100 -acodec aac -f rtsp rtsp://192.168.1.204:8554/audio
ffmpeg -f lavfi -i color=c=0x000000:s=1280*720:r=25 -vcodec h264 -f alsa -i plughw:CARD=sbj3308,DEV=0 -ac 2 -ar 44100 -acodec aac -f rtsp rtsp://192.168.1.204:8554/audio

Note: The self-built RTSP stream receiving server can be quickly built using MediaMTX (formerly rtsp-simple-server). Project address:  https://github.com/bluenviron/mediamtx

3) Capture the network stream to the local

# 将RTSP流保存为文件
ffmpeg -i rtsp://@192.168.241.1:62156 -acodec copy -vcodec copy -f mp4 c:/abc.mp4
## 注：
##	也可以重新编码以提高画质
##	ffmpeg -i rtsp://@192.168.241.1:62156 -b 900k -vcodec copy -r 60 -y MyVdeoFFmpeg.avi

# 直播录制
ffmpeg -i https://xxx/Mintimate.m3u8 -c:v copy -c:a copy -bsf:a aac_adtstoasc Output.mp4

 2.2 Use of FFplay

2.2.1 ffplay parameter description

    -i specifies the input source
    -volume specifies the number of channels
    -x specifies the screen width
    -y specifies the screen height
    -s specifies the screen width and height information (eg: 640*480)
    -f specifies the output format

    -t specifies the playback duration
    -framerate specifies the frame rate
    -fs full screen playback
    -an disables audio
    -vn disables video
    -sn disables subtitles
    -ss starts playback position
    -acodec specifies the audio playback decoder, which is equivalent to -codec:a
    -vcodec specifies video playback Decoder, equivalent to -codec:v
    -window_title lpf specifies the name displayed when the player plays the video
    -loop number specifies the number of playback loops
    -nostats does not output video-related information (you can check how ffplay collects video information through this, and write the player yourself It can be used for reference)
    -ast executes audio stream indexing
    -vst executes video stream indexing
    -autoexit automatically exits after playback is complete

2.2.2 Video playback

# 强制视频播放宽度和高度
ffplay -i test_1920x1080.mp4 -volume 1 -x 800 -y 480

# 从指定位置开始播放视频
ffplay -i test_1280x720.mp4 -volume 1 -x 800 -y 480 -fs -ss 00:05:55

# 控制视频播放流的索引
ffplay -i mult.ts -x 800 -y 480 -vst 4 -ast 3

# 指定播放时长，超过时长自动退出
ffplay -i mult.ts -x 800 -y 480 -t 5 -autoexit

# 指定视频播放解码器
ffplay -i test_1280x720.mp4 -x 800 -y 480 -t 5 -autoexit -codec:v h264

# 指定音频播放解码器
ffplay -i test_1280x720.mp4 -x 800 -y 480  -t 5 -autoexit -codec:a libfdk_aac

# 播放rtmp流媒体
ffplay -window_title "cctv1" -x 640 -y 480 rtmp://media3.scctv.net/live/scctv_800

# 原编码流播放
ffplay -i test_1280x720.mp4 -codec copy -ss 10 -t 20 -f flv out.mp4

# 播放yuv裸流视频（裸流视频播放必须指定宽度和高度、视频格式、帧率信息，其中宽度和高度不指定会绿屏或花屏）
ffplay -pixel_format yuv420p -video_size 320x240 -framerate 5 yuv420p_320x240.yuv
或
ffplay -i yuv420p_640x480.yuv -s 640*480

# 播放rgb视频（rgb格式数据播放时必须指定-pixel_format rgb24以及分辨率，否则播放不出来，不指定-pixel_format会花屏或绿屏）
ffplay -i rgb24_320x240.rgb -pixel_format rgb24 -video_size 320x240 

# 播放pcm音频（裸流视频必须指定采样率、通道数、位深信息）
ffplay -ar 48000 -ac 2 -f f32le 48000_2_f32le.pcm

 2.2.3 Play with filter

# 视频旋转播放
ffplay -i test_1280x720.mp4 -x 800 -y 480  -t 25 -autoexit -codec:a libfdk_aac  -window_title lpf -vf transpose=1

# 视频反转播放
ffplay test.mp4 -vf hflip
ffplay test.mp4 -vf vflip

# 视频旋转和反转播放
ffplay test.mp4 -vf hflip,transpose=1

# 音频变速播放
ffplay -i test_1280x720.mp4 -x 800 -y 480  -t 25 -autoexit -codec:a libfdk_aac  -window_title lpf -af atempo=2

# 视频变速播放
ffplay -i test.mp4 -vf setpts=PTS/2

# 视频和音频同时变速播放
ffplay -i test_1280x720.mp4 -x 800 -y 480  -t 25 -autoexit -codec:a libfdk_aac  -window_title lpf -af atempo=2 -vf setpts=PTS/2

# 画中画（副视频停下后默认显示最后一帧图像）
ffplay -i input.mp4 -vf "movie=sub_320x240.mp4[sub];[in][sub]overlay=x=20:y=20[out]"

# 画中画（短的视频停止后都会停止，通过eof_action控制）
ffplay -i input.mp4 -vf "movie=sub_320x240.mp4[sub];[in][sub]overlay=x=20:y=20:eof_action=1[out]"
# 画中画（短的视频停止后都会停止，通过shortest控制）
ffplay -i input.mp4 -vf "movie=sub_320x240.mp4[sub];[in][sub]overlay=x=20:y=20:shortest=1[out]"

# 画中画（短的视频停止后画面结束）
ffplay -i input.mp4 -vf "movie=sub_320x240.mp4[sub];[in][sub]overlay=x=20:y=20:eof_action=2[out]"

# 对副视频进行缩放播放
ffplay -i input.mp4 -vf "movie=sub_320x240.mp4,scale=640x480[sub];[in][sub]overlay=x=20:y=20[out]"

# 跑马灯播放
ffplay -i test_60_1280x720.mp4 -vf "movie=test_60_1280x720.mp4,scale=320x270[test];[in][test]overlay=x=mod(50*t\,main_w):y=abs(sin(t))*main_h*0.7[out]"

The scripts used by FFmpeg in this article contain various decoders that need to be installed independently. If the codec cannot be recognized (such as H264, x264, etc.) when executing the script, please install and update FFmpeg yourself (if necessary, we will organize it later) Chapter FFmpeg decoder installation).

The above is a collection of basic operations of FFmpeg. Part of the content and scripts in this article are from the network. If there is any infringement, please contact me to delete it . I hope this article can be helpful to you!