ffmpeg extension supports flv encapsulation of H265 video stream
Since Adobe has suspended updates to the RTMP/FLV standard, the HEVC video encoding format is currently not supported in the standard. In order to avoid compatibility issues between terminals and servers, FFmpeg does not implement HEVC-related extensions in the RTMP/FLV protocol implementation. The CDN Alliance has formulated relevant protocol extension specifications and completed the relevant code implementation in FFmpeg. This article mainly introduces how to extend ffmpeg to support the HTTP-FLV streaming protocol of the HEVC video encoding format.
flv protocol extension supports HEVC
For an introduction to the FLV protocol, refer to the article: HTTP FLV interaction process and example analysis
For an introduction to ffmpeg, refer to the article: A powerful auxiliary tool in the audio and video field - introduction to common operations of ffmpeg
In order to promote the implementation of HEVC video coding format in live broadcast solutions, the CDN Alliance has standardized the extension of HEVC in RTMP/FLV. In the VideoTagHeader that supports HEVC
FLV protocol extension supports HEVC
- Supports the VideoTagHeader extension of HEVC. The extended format is as shown below:
The following parts have been modified:
- CodecID: defines the value of HEVC format as 12
- HEVCPacketType: When CodecID=12, AVCPacketType is changed to HEVCPacketType; HEVCPacketType=0 means that HEVC sequence header is stored in HEVCVIDEOPACKET; HEVCPacketType=1 means that HEVC NALU is stored in HEVCVIDEOPACKET; HEVCPacketType=2 means that HEVCVIDEPACKET is stored in Is the HEVC end of sequence, that is, HEVCDecoderConfigurationRecord
- CompositionTime: When CodecID=12, CompositionTime is also required
- Supports the VideoTagBody extension of HEVC. The extended format is as follows:
When CodecID is 12, the VideoTagBody stores the HEVC video frame content
ffmpeg extension supports HEVC flv package source code modification
The demultiplexing and reusing function codes of FLV are in libavformt/flvdec.c and libavformat/flvenc.c respectively. The extended modifications are also concentrated in these two files.
- Enumeration extensions for encoding types
Modify the file libavformat/flv.h as follows, add FLV_CODECID_HEVC to CodecId:
enum {
FLV_CODECID_H263 = 2,
FLV_CODECID_SCREEN = 3,
FLV_CODECID_VP6 = 4,
FLV_CODECID_VP6A = 5,
FLV_CODECID_SCREEN2 = 6,
FLV_CODECID_H264 = 7,
FLV_CODECID_REALH263= 8,
FLV_CODECID_MPEG4 = 9,
FLV_CODECID_HEVC = 12,
};
- flv demultiplexing modification
When HEVCPacketType=0, it means that the HEVC sequence header, which is HEVCDecoderConfigurationRecord, is stored in HEVCVIDEOPACKET. When decoding, HEVCDecoderConfigurationRecord needs to be set for correct decoding. Add the AV_CODEC_ID_HEVC judgment condition where the Video Tag is read.
Modify the file libavformt/flvdec.c as follows:
if (st->codecpar->codec_id == AV_CODEC_ID_AAC ||
st->codecpar->codec_id == AV_CODEC_ID_H264 ||
st->codecpar->codec_id == AV_CODEC_ID_HEVC ||
st->codecpar->codec_id == AV_CODEC_ID_MPEG4) {
int type = avio_r8(s->pb);
size--;
if (st->codecpar->codec_id == AV_CODEC_ID_H264 ||
st->codecpar->codec_id == AV_CODEC_ID_HEVC ||
st->codecpar->codec_id == AV_CODEC_ID_MPEG4) {
// sign extension
int32_t cts = (avio_rb24(s->pb) + 0xff800000) ^ 0xff800000;
pts = dts + cts;
if (cts< 0) { // dts might be wrong
if (!flv->wrong_dts)
av_log(s, AV_LOG_WARNING,
"Negative cts, previous timestamps might be wrong.\n");
flv->wrong_dts = 1;
} else if (FFABS(dts - pts) > 1000*60*15) {
av_log(s, AV_LOG_WARNING,
"invalid timestamps %"PRId64" %"PRId64"\n", dts, pts);
dts = pts = AV_NOPTS_VALUE;
}
}
if (type == 0 &&(!st->codecpar->extradata ||
st->codecpar->codec_id == AV_CODEC_ID_AAC ||
st->codecpar->codec_id == AV_CODEC_ID_HEVC ||
st->codecpar->codec_id == AV_CODEC_ID_H264)) {
AVDictionaryEntry *t;
if (st->codecpar->extradata) {
if ((ret = flv_queue_extradata(flv, s->pb, stream_type, size)) < 0)
return ret;
ret = FFERROR_REDO;
goto leave;
}
if ((ret = flv_get_extradata(s, st, size)) < 0)
return ret;
……
}
}
3. flv reuse modification
Mainly modify the header, packet, trailer, and modify the file libavformat/flvenc.c. There are many changes. The files that need to be modified can be obtained directly from the following connection:
https://github.com/ksvc/FFmpeg
This article is also compiled and packaged based on the information they compiled.
Compilation and installation steps after modifying the ffmpeg extension
The test environment is: centos 7 x86-64
- Download x265 and compile:
wget http://ftp.videolan.org/pub/videolan/x265/x265_3.2.tar.gz
tar -zxvf x265_3.2.tar.gz
./make-Makefiles.bash
make
make install
pkg-config --list-all #查看x265是否安装成功
- Compile the modified ffmpeg
Download the source code from the github above, and the compilation process is as follows:
#下载修改后的ffmpeg，解压，https://github.com/ksvc/FFmpeg/archive/refs/tags/n3.3.2.tar.gz
# 编译步骤如下：
./configure --enable-shared --enable-libx264 --enable-gpl --enable-libx265
make&make install
ffmpeg version # 查看是否编译成功
- Test verification
Test with the following command to see if it is successful. Refer to the article "Using nginx to build an HTTP FLV streaming media server" to build the http-flv server.
Find a HEVC rtsp video connection and enter the following command to see if flv can be packaged normally.
ffmpeg -i rtsp://admin:@******@10.45.13.236/h265/ch1/main/av_stream -vcodec copy -acodec copy -f flv rtmp://127.0.0.1:1935/live/test
(Note, the open source version of nginx’s rtmp extension module does not support the HTTP-FLV protocol of H265. You need to spend money to purchase patches from the open source author. If you need to test, you can install SRS streaming media. Server, this is already supported and will be introduced in subsequent articles)
Afterword
This morning (2023-11-11) I saw that ffmpeg released the latest version 6.1, which has added support for flv h265. If you are interested, you can download the latest official version directly and try to see if flv h265 is already supported. Format, the ffmpeg6.1 version installation package can also be used. Follow the public account: 一Zero** warehouse and send a message: Software Tools obtained.
The installation steps are as follows:
tar -jxvf ffmpeg-6.1.tar.gz
./configure --enable-shared --prefix=/usr/ffmpeg
make & make install
vi /etc/profile
# 增加ffmpeg执行路径的环境变量：
PATH=$PATH:/usr/ffmpeg/bin
export PATH
# 使环境变量生效：
source /etc/profile
vi /etc/ld.so.conf
# 增加系统连接库路径：/usr/ffmpeg/lib/
# 生效
ldconfig
ffmpeg version
The latest version update instructions are as follows:
增加了 libaribcaption 解码器
增加 Playdate 视频 Decoder 和 Demuxer
增加了 afireqsrc 音频源滤镜
增加了 arls 音频滤镜
增加了 zoneplate 视频源滤镜
在 Windows 上扩展对 libva-win32 的 VAAPI 支持
在 setpts 和 asetpts 滤镜中支持命令
完善 Vulkan 解码硬件加速，支持 H264、HEVC 和 AV1
增加了 color_vulkan 滤镜
增加了 bwdif_vulkan 滤镜
增加了 nlmeans_vulkan 滤镜
增加了 RivaTuner 视频解码器
增加了 xfade_vulkan 滤镜
增加了 vMix 视频解码器
增加了 Essential Video Coding parser、muxer 和 demuxer
增加了 Essential Video Coding 帧合并 bsf
增加了 bwdif_cuda 滤镜
实现了 apsnr 和 asisdr 音频滤镜
增加了 Microsoft RLE 视频编码器
Raw AC-4 Muxer 和 Demuxer
Raw VVC bitstream parser、Muxer 和 Demuxer
增加了 用于编辑 VVC 流中metadata的bsf
将 VVC 从 MP4 转换为 Annex B 的 bitstream 滤镜
实现了 apsnr 和 asisdr 音频滤镜
实现了 videotoolbox 的 scale_vt 滤镜
实现了 videotoolbox 的 transpose_vt 滤镜
支持 P_SKIP 提示以加速 libx264 编码
在增强的 flv 格式中支持 HEVC、VP9、AV1 编解码器
实现了 OSQ复用器和解码器
在增强的rtmp协议中支持HEVC、VP9、AV1编解码器的fourcclist
实现了 CRI USM复用器
ffmpeg CLI的'-top'选项已弃用，推荐使用setfield过滤器
ffmpeg CLI选项：-readrate_initial_burst
完善了 VAAPI AV1编码器
ffprobe XML 输出模式已更改，以适应同一父元素内的多个可变字段元素
添加了ffprobe -output_format 选项，作为 -of 的别名
Note public众issue : 壹Zero仓，Transfer:flv，More flv text.