Table of contents
Requirements for protocols used in live streaming
From the perspective of interaction , streaming media is divided into on-demand (VOD) and live broadcast (LIVE)
live (LIVE): HLS, RTMP, http+MP4, http+flv, RTP+RTSP
on-demand (VOD): http+MP4, http+ flv, HLS, DASH.
From the perspective of business scenarios , let’s summarize common application solutions
Live broadcast: RTMP, HLS, http+flv
Audio and video calls: webrtc (RTP), SIP+RTP
VOD: http+MP4, http+flv, hls
IPTV: RTSP (signaling) +RTP (media)
conference TV: RTP (media) + SIP (signaling), H323 (signaling) + RTP (media)
video surveillance: national standard SIP (signaling) + RTP (media), RTSP (media) + RTP (media )
VOIP: SIP (signaling) + RTP (media)
Low latency, safe, and meet different requirements of unicast and multicast. The
streaming media protocol needs to choose TCP/UDP according to the target scenario, and then develop the application layer protocol.
How to choose TCP/UDP?
The biggest difference between TCP and UDP is:
TCP is connection-oriented and can control the connection provided by itself.
UDP is connectionless oriented and does not exercise control over the connections it provides.
TCP is suitable for applications that require reliable transmission, such as file transfer.
UDP is suitable for real-time applications, such as: IP telephony, video conferencing, live broadcast, etc.
However, in the actual usage scenario of streaming media, if UDP is directly used to transmit audio and video, extreme situations such as packet loss and out-of-order unavoidable in public network transmission will cause the client to fail to decode. However, if TCP is used to transmit audio and video directly, problems such as TCP's own congestion control mechanism, large data transmission delay, and blockage at the head of the line will cause certain problems to the real-time transmission of audio and video.
Therefore, for streaming media transmission, high-level development of the transport layer TCP/UDP protocol is required to meet the application requirements of streaming media transmission.
RTP
What is the RTP protocol?
RTP (Real-time Transport Protocol) is a transport layer protocol for multimedia data streams on the Internet.
The RTP protocol is used together with the RTP control protocol RTCP, and it is built on the UDP protocol.
Like UDP, it does not provide any transmission reliability guarantee and traffic congestion control mechanism, and cannot guarantee the service quality of real-time services. Here is a question, since both UDP and RTP are transport layer protocols, and RTP is above UDP, what is the relationship between the two?
In simple terms, both the RTP protocol and UDP work together to complete the transport layer protocol transmission. UDP is only responsible for transmitting data packets, and RTP provides timestamps and other technologies to ensure the time accuracy of streaming media in real-time transmission.
Why use the RTP protocol?
(1) Low latency
(2) Multicast function Multicast
is widely used in network video conferencing
. It is mainly used in such an environment . Assume that the red circle is a streaming server that stores video data, and other The circles are the clients connected to the server. When all the green clients request to watch a certain video on the red server at the same time, it is obviously unreasonable if the server establishes a separate connection for each client for data transmission. Waste of bandwidth, therefore, multicast technology can solve this problem very well, that is, the same piece of data is sent to a public multicast address by the server , and each client listens to the same multicast address to obtain data. It not only improves the bandwidth, but also ensures the synchronization of the videos watched by each client.
The RTP protocol was originally born to implement similar video conferencing applications
(3) Security, supports encrypted data and authentication functions
RTCP
What is the RTCP protocol?
The Real-time Transmission Control Protocol (Real-time Control Protocol, RTCP) and RTP are jointly defined in RFC 1889 proposed in 1996, and are control protocols that work with RTP. RTCP runs independently on the low-level protocol , and the low-level protocol provides multiplexing of data and control packets. During an RTP session, each session participant periodically sends RTCP control packets to all other participants, as shown in the figure below.
For RTP sessions or broadcasts, a single multi-destination broadcast address is usually used. All RTP and RTCP packets belonging to this session use this multi-destination broadcast address. RTP packets can be distinguished from RTCP packets by using different port numbers. .
Why use RTCP protocol?
RTCP is responsible for managing the transmission quality. During the RTP session, each participant periodically transmits RTCP packets, which contain statistical information such as the number of sent data packets and the number of lost data packets. The server uses this information to dynamically change the transfer rate and even change the payload type.
The combination of RTP and RTCP can optimize the transmission efficiency through effective feedback and minimum overhead, so it is especially suitable for transmitting real-time data on the Internet.
RTSP
What is the RTSP protocol?
RTSP, which has one more S than RTP, is the RealTime Streaming Protocol real-time streaming protocol . It is an application layer protocol above the transport layer (RTP is the transport layer). UDP, multicast UDP, TCP, and RTP can be selected as the transmission mechanism. RTSP defines how two-way multi-applications can efficiently transmit multimedia data over IP networks.
RTSP acts as a network remote control of a multimedia server, making it possible to fast forward, rewind, stop, and play real-time data such as audio and video.
Compared with RTP, RTSP is a two-way real-time data transmission protocol in the application scenario, which allows the client to send requests to the server, such as playback, fast forward, rewind and other operations. In terms of protocol, RTSP is an application layer protocol, which can transmit data based on RTP, and can also choose TCP, UDP, multicast UDP and other channels to send data, which has good scalability. Here it is relatively clear. The IPTV live broadcast service of the operator does not have any operations such as playback and rewinding, so UDP+RTP+multicast can be directly implemented.
Why use RTSP protocol?
Control to the video frame, so it can host applications with high real-time
RTMP
What is the RTMP protocol?
RTMP (Real Time Messaging Protocol) is an open protocol developed by Adobe Systems for audio, video and data transmission between Flash players and servers.
The RTMP protocol belongs to the application layer and is designed to multiplex and package multimedia transport streams (such as audio, video and interactive content) on a suitable transport protocol (such as TCP). RTMP provides a set of full-duplex reliable multiplexing message services, similar to the TCP protocol [RFC0793], used to transmit time-stamped audio streams, video streams, and data streams in parallel between a pair of nodes. Usually, different types of messages are assigned different priorities. When the network transmission capacity is limited, the priority is used to control the queuing order of messages at the bottom layer of the network.
Why use RTMP protocol?
Low latency, high stability, support for all camera formats, the browser can load the flash plug-in to play directly
Summary of RTP, RTCP, RTSP
- RTP provides time stamps, sequence numbers, and other methods that can guarantee processing time during real-time data transmission.
- RTCP is the control part of RTP, which is used to guarantee the quality of service and member management.
- The specific data transmission of RTSP is handed over to RTP to provide remote control of the flow.
HLS
What is the HLS protocol?
HTTP Live Streaming (HLS) is an HTTP-based streaming media transmission protocol implemented by Apple Inc., which can realize live streaming and on-demand streaming media. It is mainly used in iOS systems to provide audio for iOS devices (such as iPhone and iPad). Live video and on-demand solutions. HLS on-demand is basically the common segmented HTTP on-demand, the difference is that its segments are very small.
Compared with common streaming media live broadcast protocols, such as RTMP protocol, RTSP protocol, MMS protocol, etc., the biggest difference of HLS live broadcast is that what the live broadcast client obtains is not a complete data stream. The HLS protocol stores the live data stream as continuous, short-duration media files (MPEG-TS format) on the server side , and the client side continuously downloads and plays these small files, because the server side always sends the latest live broadcast The data generates new small files, so that the client only needs to continuously play the files obtained from the server in sequence, and the live broadcast is realized. It can be seen that, basically, it can be considered that HLS implements live broadcasting by means of on-demand technology. Since the data is transmitted through the HTTP protocol, there is no need to consider firewall or proxy issues at all, and the duration of segmented files is very short, so the client can quickly select and switch bit rates to adapt to playback under different bandwidth conditions. However, the technical characteristics of HLS determine that its delay is generally higher than that of ordinary live streaming protocols.
Why use the HLS protocol?
(1) Cross-platform: support iOS/Android/browser, strong versatility.
(2) Strong ability to pass through walls: Since HLS is based on the HTTP protocol, HTTP data can pass through firewalls or proxy server HLS can do it, and basically will not encounter the situation of being blocked by the firewall.
(3) Fast bit rate switching (definition): With self-adaptive multi-bit rate, the client can choose to download the same resource at different rates from many different alternate sources, allowing streaming sessions to adapt to different data rates. The client can quickly select and switch bit rates to adapt to playback under different bandwidth conditions.
(4) Load balancing: HLS is based on a stateless protocol (HTTP). The client only uses and downloads ordinary TS files stored on the server in order. The load balancing is as simple as the ordinary HTTP file server.
A new generation of live transmission protocol SRT
What is the SRT protocol?
Secure Reliable Transport (SRT) is an open source Internet transmission protocol based on the UDT protocol. Haivision and Wowza jointly established the SRT Alliance, an organization that manages and supports open source applications of the SRT protocol. This organization is dedicated to promoting video streaming solutions The interoperability of solutions, as well as the promotion of cooperation among pioneers in the video industry, to achieve low-latency network video transmission.
SRT is a very popular open source low-latency video transmission protocol nowadays. SRT solves complex transmission timing problems. SRT can reduce delay, eliminate central bottlenecks, and reduce network costs.
Why use the SRT protocol?
(1) Reliability: Adapt to any network environment, efficiently handle network packet loss, jitter and bandwidth fluctuations and other interferences
(2) Low delay: Due to the use of UDP transmission mode and the use of ARQ packet loss recovery mechanism, based on the public network The transmission delay level can generally be controlled within 1s
(3) High quality: SRT's transmission and error correction mechanism can maximize the use of available bandwidth and eliminate network errors and interference, so higher bit rate video streams can be transmitted in the same network environment , with high-efficiency coding formats such as H.264 and HEVC, it can still guarantee high-quality video under bad network conditions
(4) High bandwidth utilization: different from ABR, the multi-bit rate adaptive distribution technology needs to be a redundant bit rate Occupying additional bandwidth, SRT monitors the network link status in real time, and can perform real-time bit rate adjustment (NAE, Network Adaptive Encoding). In addition, compared with TCP's packet loss recovery mechanism, ARQ's packet loss recovery mechanism also greatly saves bandwidth and reduces network congestion
(5) Security: SRT uses AES-128 or 256 encryption to protect content security
(6) Free and open source: SRT is completely free open source
Multicast, broadcast and unicast
TCP is a connection-oriented protocol. TCP must be point-to-point, and two hosts must establish a connection. TCP must be unicast .
Only UDP uses broadcast and multicast.
Sometimes a host wants to send frames to all other hosts on the network. This is broadcasting. Broadcasting is divided into Layer 2 broadcasting (the destination MAC is all F) and Layer 3 broadcasting (the host bits of the IP address are all 1). Layer 2 broadcasting cannot cross routers. Yes, Layer 3 broadcasts can be routed across routers.
Multicast (multicast) falls between unicast and broadcast, frames are delivered only to multiple hosts belonging to the multicast group.
In the field of broadcasting and television, in order to reduce server pressure, multicast is usually used to push streams with users. Such as IPTV, usually the set-top box joins a certain multicast address through the optical modem, and receives the multicast stream of a certain CDN.
reference content
http://www.52im.net/thread-267-1-1.html
https://zhuanlan.zhihu.com/p/496823042
https://blog.csdn.net/fengliang191/article/details/120813885
https://blog.csdn.net/weixin_38054045/article/details/104390238
https://zhuanlan.zhihu.com/p/551804049
https://cloud.tencent.com/developer/article/1038381
https://blog.csdn.net/xiaoliuliu2050/article/details/103954804
https://zhuanlan.zhihu.com/p/114436399
https://zhuanlan.zhihu.com/p/403771996