Video image transmission has the following characteristics
1: Requirements small transmission delay, real-time high
2: transport flow, high transmission efficiency is required
3: allowing a certain program transfer error or data loss
Using the UDP protocol to transfer video relatively more preferably the TCP protocol ( barabara UDP and TCP difference )
With a simple phrase summed up: RTSP origination / termination streaming, RTP streaming data transmission, RTCP for RTP control, synchronization . The reason why before these few little hard to tell, because the CTC there are no standards for RTCP be required , and therefore the standard did not see the relevant section RTSP code. In the private RTSP code, relating to control, synchronization, etc., are doing extended definition implemented in the RTP Header .
RTP: Real-time Transport Protocol (Real-time Transport Protocol)
RTP / RTCP is a protocol for transmitting data real
RTP transport audio / video data, if the PLAY, is sent to Client-Server, if the RECORD, may be sent from the Client to the Server
The entire RTP protocol consists of two closely related composition: data protocol RTP and RTP Control Protocol (i.e. RTCP)
RTSP: Real Time Streaming Protocol (Real Time Streaming Protocol, RTSP)
RTSP requests mainly DESCRIBE, SETUP, PLAY, PAUSE, TEARDOWN, OPTIONS, etc., by definition can know from the conversations and control action
RTSP dialogue process can be determined in the SETUP port RTP / RTCP use, PLAY / PAUSE / TEARDOWN can start or stop sending RTP, and so on
RTCP：
RTP / RTCP is a protocol for transmitting data real
Including RTCP Sender Report and Receiver Report, used for synchronization purposes and other audio / video, which is a control protocol
Summary description of each protocol :
A, the RTP data protocol
RTP protocol is responsible for streaming media data packet and the data transmission of real-time media stream, each of the two parts by RTP datagram header (Header) and a load (Payload), where the front head 12 bytes meaning fixed, and the load can be an audio or video data. RTP datagram header format shown in Figure 1:
The more important ones domain and its significance as follows:
CSRC count (CC): indicates the number of CSRC identified. RTP CSRC identifier immediately after the fixed head, to indicate the source of the RTP data packets, RTP protocol allows multiple data sources in the same session, they may be combined into one data source through an RTP mixer. For example, you can generate a CSRC list to represent a conference call, the conference RTP mixer through a combination of all the voice data to a speaker RTP data source.
Payload type ( the PT): RTP payload format indicated, including the encoding algorithm used, the sampling frequency, the carrier channel and the like. For example, type 2 indicate that the RTP data packets is carried in ITU G.721 algorithm encoded voice data, the sampling frequency is 8000Hz, and a single channel.
Serial Number: to provide a method of detecting data loss as a receiver, but how to deal with lost data is the application's own thing, the RTP protocol itself is not responsible for the retransmission of data.
Timestamp: Record the load in the first byte of the sampling time, the receiver can determine the arrival time stamp data can be affected by whether the delay jitter, but the specifics of how to compensate for delay jitter is the application's own thing.
Easy to see from the RTP datagram format that contains the transmission media types, formats, sequence number, time stamp, and whether the additional information data, which are real-time streaming media to provide the corresponding base. RTP protocol object is to provide real-time data (e.g., audio and video interactive) end to end transport service, and therefore the concept of the RTP is not connected, it can be built on top of non-connection oriented or connection-oriented transport protocol underlying ; RTP does not depend on a particular network address format, and only need to support framing of the underlying transport protocols (framing) and the segment (segmentation) is sufficient; RTP itself does not provide any additional reliability mechanisms, these must be made transport protocol or application itself to be guaranteed. In a typical application, the RTP protocol is generally above the transport as part of the application to be implemented, Figure 2:
Two, the RTCP control protocol
RTCP control protocol need to be used together with RTP data protocol, when the application starts a RTP session will also occupy two ports for RTP and RTCP are used. RTP itself does not provide for the transmission of data packets in sequence a reliable guarantee, does not provide flow control and congestion control, which is responsible for completion by RTCP. RTP and RTCP typically will use the same distribution mechanism, transmit control information to all members of the session periodically, by receiving application data from which to obtain relevant information session participants, and network status, packet loss probability Feedback , it is possible to control the quality of service or network status diagnosis.
RTCP protocol function is achieved through different RTCP packets, there are the following types:
SR: sending end report, the term refers to the transmitting side RTP data packets sent to the application or the terminal, the transmitting side also can be a receiving end.
RR: report receiving end, the receiving end refers to the so-called only receive but not transmit RTP data packet or a terminal application.
SDES: Source description, the main function is as a session member identification information about the vehicle, such as user name, mailing address, telephone number, etc., in addition to having a session control function to convey information to the session members.
BYE: notice to leave, the main function is to indicate one or several sources are no longer valid, notify other members of the session that he will exit the session.
APP: defined by the application itself, solve the scalability problem RTCP and provides great flexibility for the realization of those agreements.
RTCP packets carry the necessary information to monitor the quality of service can dynamically adjust the quality of service, and be able to effectively control of network congestion. Since RTCP packets is used in multicast mode, so that all members of the session information can be controlled through RTCP packets returned, to understand the current situation of the other participants.
In a typical application, the application will send the media stream periodically reports the transmission side generates the SR, the RTCP packet comprising synchronization information data between the different media streams and a count of bytes and packets have been transmitted, received end may be estimated in accordance with the actual data transmission rate information. On the other hand, the receiving side sends the transmitting end receives all known clients report RR, the RTCP data packet containing the maximum sequence number of packets received, the number of data packets critical information loss, delay and jitter timestamp, the transmitting side application information can be estimated round trip delay and loss probability and delay can be dynamically adjusted based on the data jitter packet transmission rate, in order to improve network congestion, or to smoothly adjust the quality of the service application according to the network conditions.
Three, RTSP Real Time Streaming Protocol
As an application layer protocol, RTSP provides a framework for expansion, its significance lies in making a controlled on-demand and real-time streaming data becomes possible. Overall, RTSP is a streaming media that protocol which is used to control the data transmission having real-time characteristics, but does not itself transmit data, but must rely on certain services provided by the underlying transport protocol. RTSP may provide streaming media such as play, pause, fast-forward operation, which is responsible for the definition of a specific control message, method of operation, status code, etc., in addition to the operation described in connection with the interaction between the RTP (RFC2326).
RTSP reference in the development of many of the HTTP / 1.1 protocol, even many description and HTTP / 1.1 are identical. RTSP reason deliberately using similar syntax and operations with HTTP / 1.1, largely for compatibility with existing Web infrastructure, and as such, the extension mechanism of HTTP / 1.1, most can be introduced directly into the RTSP.
The RTSP media control stream collection may be defined by a presentation description (Presentation Description), is a collection of so-called represents one or more streaming server to the client of the media stream, and a presentation description contains a representation of each media information flow, such as data encoding / decoding algorithms, network addresses, and other media content stream.
Although RTSP server also use the identifier to distinguish each flow connection session (the Session), but RTSP connection and is not bound to the transport layer connection (e.g., TCP, etc.), that is to say during the whole RTSP connection, the user can open or RTSP Close plurality of reliable transport connections to the server to issue RTSP RTSP request. Further, RTSP connection can also be based on a connectionless-oriented transport protocol (such as UDP, etc.).
RTSP protocol currently supports the following operations:
Retrieving media: allows the user to submit to the HTTP server, or other media presentation description method. The representation is multicast, the presentation description contains a multicast address and port number of the media stream; if unicast is represented, for security purposes should be provided only in the address represented by the description.
Invite: The media server can be invited to a meeting in progress or playback of media representation, or record all or a subset of the media in the representation, ideally suited for distributed teaching.
Add media: to inform the user can take advantage of the newly added media stream, which is especially useful in terms of on-site lectures. And similar to HTTP / 1.1, RTSP request may be handed over to agents, or channel buffer for processing.
Difference between HTTP and RTSP transmission. General speaking, RTSP is denied many corporate firewalls, and HTTP as an ordinary file by; RTSP suitable for large amounts of data, high availability streams, such as live events, event long or large file; HTTP is more suitable for smaller and interactive data transmission; when an end user is viewing, RTSP server allows the user to valid media playback, HTTP download section more like a media player on the client and. , RTSP looks like a file to play from the end user point of view from a central location, a bit like broadcasting, and video taken from the video library feel more like an HTTP, and played on home machines. From the point of view of service quality point of view, for streaming, RTSP have a better experience, RTSP provides VCR like control of the media, such as pause, fast forward, rewind, and absolute positioning. Using the HTTP transport, the entire stream only after the download is complete, then the player software simulation of the process. Although, RTSP can use TCP or UDP, RTSP control but is often used in conjunction with RTP, the best quality service delivery actual media data.
There are many video-related protocols, different companies, and even has its own protocol standards. This article try to cover the current common video related agreements.
1, RTSP / RTP / RTCP protocol suite.
This protocol suite is the first video transmission protocol. Wherein the RTSP protocol is used for video on demand session control, e.g. demand request initiates a SETUP request, the PLAY specific playback operation, the PAUSE request, the parameter is also supported by the video jump PLAY request. RTP protocol for the transmission of a particular video data stream. The RTCP protocol is a control means C, in addition to the video stream data, or a control packet loss rate or the like.
The protocol family RTSP is built on top of TCP, RTP, RTCP built on top of UDP. But there may be together in the same connection on a TCP transport interleave manner by RTP and RTSP.
RTSP protocol suite, is that so many places to consider the first to be put forward, still with the early features. Such as the use of UDP, taking into account the efficiency of the transmission and video protocols themselves have a certain tolerance for packet loss. But the UDP protocol, obviously can not be used for larger networks, but also to penetrate the problem under complex network router.
From the perspective of the video, the advantages of RTSP protocol suite that can be controlled to a video frame, it is possible to carry real-time high application. This advantage is the biggest advantage over the HTTP method. H.323 video conferencing protocol, the underlying general use RTSP protocol. RTSP protocol complexity Group focused on the server side, because the video file server needs to parse, Seek to a specific video frame, but may also need to be double-speed playback (that is, the old DVD with the kind of double speed, 4-speed playback function), double-speed playback function is unique to the RTSP protocol, other protocols can not support video.
Drawback is the complexity of the server side is relatively high, it is also more complicated to achieve.
2, HTTP protocol
HTTP video protocol, mainly after the popularity of the Internet. Internet point of view of the formation of the lower demand video.
2.1
The original HTTP protocol video, there is nothing about, is a common file HTTP progressive download. The essence is to download video files, and use of the characteristics of the video file itself, is the existence of header information, and a portion of the video frame data, they can decode the broadcast. Obviously, this approach requires the header information of the video file in front of the file. Some example faststart tool is designed to do this function.
But under the most primitive state, the video can not play fast forward or jump to the file has not been downloaded to the part. This time of the HTTP protocol proposed range-request requirement. This almost all HTTP servers are supported. range-request, the request is part of the data file, the offset specifies the number of bytes. After parsing the client video head of the video file, you can determine the location of a respective subsequent frame of video. According to information bit rate, or the like, is calculated the corresponding position.
2.2
Supports HTTP range-request server, it can currently support, we generally see network video request. However, this approach can not support real-time video streaming, or near real-time video streaming. Because the video stream, is not a concept video files exist. HTTP protocol benefits of video playback, is simple. Take generic HTTP server, you can provide the service, does not require a special type of server and does not require special network ports through the firewall router no problem at all. But also can be used to simplify the work of universal CDN deployment of video distribution, reduce the use of bandwidth.
This is a PC client or web client, video-on-demand services, the most common solution. Client implementation, commonly used flash is completed, this is a flash or videoplayer videodisplay control can be completed. Resources commonly used formats flv, mp4 format can also be used.
On this basis, a number of companies launched their own solutions.
2.3 HLS HDS MSS DASH
Apple's HTTP Live Streaming, is a kind With the popularity of Apple devices can be widely applied. From the noun can be judged, HLS Live support live video transmission type. M3u8 using HTTP as the index file, the video format is MPEG2-TS clip file. If a live video stream, then take m3u8 update file to update the list of video indexing, achieve the purpose of live video. This method, however, because the final video file a fragment, there must be a delay so that the length of the video clip. Therefore only be used for not so high near real-time video streaming of real-time requirements. But HLS way, because the use of the earlier MPEG2-TS format, overhead this format, that is, the proportion of the total file header information occupies a relatively large, that is not efficient. Why not use other formats, mainly commercial competition and copyright issues.
Appropriate, Adobe company, launched a similar HTTP dynamic streaming. In this way nature and HLS strategy is similar, that is, by the way the index file + video clips. But apparently used an index format and video clips are not the same format. HDS is used in video format is flv or f4v, index format remember.
Similarly, Microsoft also introduced the Microsoft Smooth Streaming, that is, mss video broadcast mode, using the video format is segmented mp4 format.
The MPEG standard group is introduced Dynamic Adaptive Streaming over HTTP, DASH. 3GPP video formats to use it.
Obviously, this is now flourishing state, although the most common is the HLS, who told Apple so domineering it. The Android and Apple for flash above have limitations, and HDS are available in adobe adobe platform above. mss, only seen in a few places. dash, currently only the paper inside the product.
Above agreements, mainly to resolve a problem, adaptive rate is switched, the above dynamic, adaptive all similar meaning. Mainly address the use of the index file and gives a different bit rate video files, so that the player terminal, according to bandwidth, selecting different rate adaptive video file.
At the same time in terms of live video, Andrews and beyond iOS platform, there are many video server vendors to develop their own protocol, commonly used method is the way flv files fixed time segments (using flv, is because it's easy to play on flash)
2.4 HTML5
Meanwhile HTML from manufacturers not lonely, launched the HTML5 in this way, in essence, in this way, and in front of the HTTP protocol video without any distinction. But the player side, no longer tied to a particular plug-ins such as flash, or other playback software, but can support direct video player browser. Is used in embedded video html tags were directly identified video url. However, different browsers, not exactly the same for audio and video format support. More common is the video H. 264 formats, audio AAC, encapsulation format MP4.
2.5 Other
In the HTTP protocol, based on each company for their business characteristics, they may also develop their own proprietary data format or protocol. However, all the above is basically to make some minor modifications only. After all, the domestic technology system, also completely unable to put forward a new level of technology. (Google, replace H.264, SPDY protocol proposed new encoding format, this country can not do).
Usually only done e.g. HTTP special fields or special parameters passed to the flash to make some explanations. Or add some special significance of the head, etc. in the original video file formats on.
3，RTMP
This is the adobe company launched its own video playback agreement. Requires a dedicated server, such as FMS, there are open source red5. This protocol is also supported by default flash.
----
Overall, the video-related protocol development, from specialized protocols, servers, and gradually to HTTP transition process. And gradually adapt to the network's development and needs of complex network environment, it gave birth to the video http protocol.
1, the video bit rate is generally set much?
For 1080P video, the Blu-ray video bit rate is 20Mb / s, generally downloaded video bit rate mostly 10Mb / s, some of IPCamera / UAV rate is 2 ~ 8Mb / s, and many video sites rate even lower than 5M / s
Case of the same resolution, the higher the rate, the greater the clarity, but the network bandwidth is greater, the specific rate set how much need to see a specific application scenario.
"Jump" and "Huaping" phenomenon 2. Play appear?
"Jump" and "Huaping" phenomenon because the vast majority of network signal transmission process due to bad result in the loss of the "key frame" / "reference frame" caused further explained below.
Before the spread of online video compression is required, and simple to explain the video compression core idea is this: every 10 to 50 to take the video image as a "key frame", and the next few frame images due to time / space redundancy and correlation, we just record it with "difference" information can be, so the video file keyframes you can not complete each frame of image data are all saved, which played a space-saving effect.
Thus, if you lose out "key frame", the next few natural frame image can not be properly decoded, thus creating a "Huaping" phenomenon.
From a technical point of view, how to solve the "Huaping" phenomenon? - When we in the video transmission process, dropped frames by frame number found, then skip the non "key frame", until the next key frame is subdivided into decoded. This can indeed solve the "Huaping" phenomenon, but due to jump a lot of frames, it is not the case of continuous video image will appear (ie "jump" phenomenon).
"Caton" phenomenon 3. appear during playback?
Since the network is very unstable, and therefore, the transmission of audio and video data is also faster or slower when, during playback of the video stream in the network, must be determined according to the time stamp when the display when decoded, instead of a to play an addition, adding a certain amount of "frame buffer" can be effectively reduced due to network jitter caused by "Caton" phenomenon.
4. Where real-time audio and video transmission delays come from?
( 1) encoder / decoder buffer typically require 2 to 4
( 2) the encoding / decoding time consuming
( Frame buffer 3) traffic code
( 4) the network transmission delay
( Copy of data 5) code
In general, the frame rate of 30f / s video, each frame buffer n, increases 1000 30 * n ms delay /. Therefore, to reduce latency, the delay must be found by analyzing each of the above and the test section, and to minimize the buffered data copy.
5. side bottom sowing of principle?
Side bottom sowing and play local files are actually similar, but is not on the local data file, before the player plays to the specified location, background thread need to download the data down in advance only.