Hi everyone, and welcome to the Stop Refactoring channel.
In this issue, we discuss in detail the related protocols of live broadcast , including: HTTP-FLV, HLS, RTMP, Web-RTC, RTSP and so on.
We will introduce in detail how these protocols work, their application scenarios, and the reasons for the delay.
We discuss in this order
1、 RTMP、HTTP-FLV
2、 HLS
3、 Web-RTC
4、 RTSP
RTMP, HTTP-FLV protocol
Both RTMP and HTTP-FLV are based on FLV encapsulation.
RTMP is generally used for live streaming , and HTTP-FLV is generally used for live viewing .
Let's discuss RTMP first. The RTMP protocol is a protocol that can both push and pull streams .
The RTMP address starts with rtmp://, and the streaming address is the same as the playback address.
However, because the browser has abandoned the Flash player, and it is said that there may be some unstable problems under high concurrency, RTMP is generally only used for scenarios such as streaming from live sources and streaming to live CDNs .
The RTMP protocol requires specific streaming service software , such as SRS, Nginx with RTMP plug-ins, etc.
As discussed in the working principle of live broadcast in the past, this kind of streaming media service software is actually a transfer station for audio and video data. Generally, the data is only cyclically overwritten in the memory and will not be written to the disk.
The delay of the RTMP protocol is relatively low, about 1-3 seconds.
RTMP communication is established on the TCP long connection channel. When encapsulating audio and video data, slices are forced to limit the size of each data packet.
Mandatory slicing also guarantees real-time performance to a certain extent. There is a certain ability to resist weak networks, because each data packet will not be too large, so when a data packet fails to verify, the cost of resending will not be too large, but also because merging data packets will increase CPU pressure , so there is a certain performance consumption.
There are also some variants of the RTMP protocol, such as RTMPT, RTMPS, etc., which will not be discussed here.
Let's discuss the HTTP-FLV protocol again.
The address starts with http://, which is based on the HTTP protocol. HTTP-FLV can be simply understood as the HTTP protocol version of RTMP . The functions and working principles are similar, and the above-mentioned RTMP slice data function HTTP-FLV is also available.
However, the HTTP-FLV protocol can generally only be used for streaming viewing .
The delay of the HTTP-FLV protocol is also relatively low, about 1-3 seconds , but the actual experience shows that the delay of HTTP-FLV is slightly higher than that of RTMP, but HTTP-FLV is more suitable for playing scenarios than RTMP.
The HTTP-FLV live stream generally needs to be played by adding a plug-in. For example, the webpage needs to import flv.js before the browser can play it. HTTP-FLV live streaming, here we need to thank the open source flv.js of station B, which promotes the popularization of HTTP-FLV in browsers.
The HTTP-FLV protocol requires specific streaming service software , such as Nginx with the HTTP-FLV plug-in.
It is worth mentioning that the HTTP-FLV plug-in of Nginx includes RTMP function, so in general HTTP-FLV streaming media service, the push stream uses the RTMP protocol, and the pull stream uses the HTTP-FLV protocol.
The more popular solution now is that the live broadcast source pushes the stream using the RTMP protocol, and the live streaming streaming watch uses the HTTP-FLV protocol.
HLS protocol
The HLS protocol is generally only used for streaming viewing, but strictly speaking, the HLS protocol is not a streaming protocol.
It works simply by downloading static files via the HTTP protocol .
The difference is that the file of the HLS protocol consists of two parts , one is multiple .ts fragmented video files with a length of only a few seconds, and the other is the .m3u8 index file that records the addresses of these video files , and these static files are written directly into disk.
More specifically, the HLS viewing address starts with http:// and ends with .m3u8. In fact, this address is the address of the index file. After the client obtains the index file, it can download the corresponding fragmented video file and start playing it. up.
Since the HLS protocol actually requests files through the HTTP protocol, and HLS-related files are directly written to the disk, no special streaming service software is required , and HTTP services such as Nginx are sufficient .
The HLS protocol can be used for on-demand and live viewing . It adapts to a variety of playback scenarios. Generally, it can be played by adding a plug-in. For example, a web page can be played by adding an HLS js plug-in. Apple devices natively support the HLS protocol.
In the on-demand scenario , that is, in the scenario of ordinary online video viewing.
The .m3u8 index file will record the addresses of all fragmented video files. HLS has more obvious advantages in on-demand scenarios .
Since the related files of HLS are stateless static files, and the size of each file is limited, the effect of load balancing and CDN acceleration is more obvious.
On-demand videos of the HLS protocol will be played faster than .mp4 and .flv videos, and video jumps during loading will be smoother.
In the live broadcast scenario , the transcoding software can directly generate HLS-related files to the disk, and the client can download the files through the HTTP service.
In addition, an RTMP plug-in can also be added to Nginx, and the transcoding software pushes the stream to Nginx through the RTMP protocol, and then Nginx generates HLS-related files.
Among them, the latter solution is more recommended , because it is more smooth for the transition of the early stage R&D and the post-docking live CDN.
In addition, the HLS-related files in the live broadcast scene are somewhat different from those on demand .
The video stream data will be packaged into a fragmented video file with the suffix .ts every few seconds, and the .m3u8 index file will be updated synchronously every time a new video file is generated .
And there is an upper limit for the number of fragmented video files . When the upper limit is reached, the oldest video file will be deleted by default and the .m3u8 index file will be updated.
Therefore, in the live broadcast scenario, the client also needs to constantly re-acquire the .m3u8 index file.
The HLS protocol has no advantages in live broadcast scenarios.
Although the live stream of the HLS protocol can also adapt to many playback scenarios, due to the need to generate static files, the live broadcast delay is very large , about 5-30 seconds . If you use a live CDN, due to edge node synchronization and other issues, the live broadcast delay may even be possible. It will reach about 1 minute.
Of course, the HLS protocol also has certain advantages. In live broadcast time shift, that is, live broadcast to on-demand, or recorded broadcast, that is, on-demand to live broadcast, theoretically only need to modify the index file.
In addition, the .m3u8 index file of the HLS protocol supports secondary indexing , that is, multiple viewing addresses such as high-definition, standard definition, and smooth can be integrated into one index file. The player can automatically switch between different viewing addresses according to the current bandwidth, and most web players' "automatic" is also because of this.
Here is a small knowledge point of the HLS protocol.
Since the video files and index files of the HLS protocol are written directly to the disk, if multiple live streams are processed simultaneously for a long time, it will cause excessive writing pressure on the disk , the mechanical disk may be damaged, and the life of the solid state drive will accelerate decay.
In this case, it is best to mount a section of memory space as the writing location of HLS-related files, which will not cause excessive disk writing pressure.
As a supplement, the HLS protocol is a standard introduced by Apple. Similar to the HLS protocol, there is also the MPEG-DASH protocol. The working principles of HLS and MPEG-DASH are similar, but some standards are different, so I won’t expand here.
WebRTC protocol
The WebRTC protocol is not actually designed for live broadcast scenarios. WebRTC is a point-to-point video/voice call protocol.
Since WebRTC is based on UDP, after the communication is established, data will be continuously sent in a stream, so the delay will be lower than RTMP.
In some highly interactive live broadcast scenarios , such as live broadcast with goods, etc., WebRTC will be used as the streaming and viewing protocol. The delay of WebRTC can theoretically reach within 1 second.
The WebRTC protocol supports push and pull streams. The address generally starts with webrtc://, and the push and pull stream addresses are generally the same.
Although WebRTC is a point-to-point protocol, if it is applied in a live broadcast scene, it is necessary to build a WebRTC server as a streaming service , and the streaming service software can use SRS.
By the way, SRS is a relatively popular open source streaming media service software developed in China. At present, 4.0 has included mainstream protocols such as RTMP, HLS, WebRTC, and HTTP-FLV.
RTSP protocol
RTSP is generally not used for live broadcast scenarios , and RTSP is generally used for viewing and pushing real-time video streams of hardware devices such as cameras and monitoring.
Although the RTSP protocol also supports push/pull streams, and supports TCP, UDP switching, and many other advantages.
But the versatility is insufficient, especially the current browsers do not support RTSP playback.
Summarize
The above is an introduction to commonly used live broadcast protocols. The delays mentioned are pure communication delays . If you want to look at the entire live broadcast process, the delay will be further amplified.
Because the live broadcast delay includes push delay, transcoding delay, and pull delay, even if WebRTC is used as the push and pull protocol, the final delay will be a few seconds.
As for the problem of live broadcast delay, although the above agreements play a key role, they often do not play an absolute role.
The reduction of live broadcast delay will involve many problems. For example, detailed issues such as prohibiting B frames, GPU hardware acceleration, caching I frames in streaming media services, bit rate restrictions, etc., will be discussed in detail later.