The most detailed audio and video streaming media transmission protocol - detailed explanation of rtsp protocol
Detailed explanation of streaming media transmission protocol-rtsp protocol
See: Detailed Explanation and Analysis of RTSP Protocol Writing an RTSP Server from Scratch (1) Explanation of RTSP Protocol An in-depth preliminary introduction to RTSP_RTP_RTCP protocol
rtsp
Before the emergence of RTSP, the hottest protocol was probably the HTTP protocol. Imagine that when you need to enjoy a certain video on the Internet, access its URL through the HTTP protocol, start downloading, and play it after the download is complete. For early video capture devices, network bandwidth, or displays responsible for rendering, it seems that a little more patience, multiple disconnected HTTP connections, and even checking the integrity of the downloaded files several times, the experience is not good. Still passable. After all, the resolution, frame rate, and bandwidth at that time limited the size of media files transmitted through the Internet, and information sharing could only be transmitted in the form of stored files through various hard drives, U disks, and CDs.
With the development of hardware equipment technology, the resolution of acquisition equipment is increasing, the display supports higher frame rates, and the network bandwidth is also increasing exponentially , all of which provide basic support for a better viewing experience. With the increasing abundance of network resources, the scarcity of users' time has become increasingly prominent. In order to quickly watch and judge whether the video itself meets their own tastes, online real-time viewing has become a major demand. The traditional HTTP download obviously cannot meet this demand, so RTSP stands out on the road of seeking streaming.
RTSP stands for Real Time Streaming Protocol (Real Time Streaming Protocol), which is a network control protocol designed to control streaming media servers in entertainment and conference systems. RTSP is used to establish and control a media session (session) at both ends of the desired communication, and the client can control the media stream in real time by issuing VCR-style commands such as play, record, and pause. You can refer to RTSP 2326 Chinese version
what is rtsp
The RTSP protocol works in a client-server manner, such as: pause/continue, back, forward, etc. It is a multimedia playback control protocol, which is used to enable users to control when playing real-time data downloaded from the Internet, so RTSP is also called "Internet Video Recorder Remote Control Protocol".
RTSP (Real-Time Stream Protocol) is a text-based application layer protocol. In terms of syntax and some message parameters, the RTSP protocol is similar to the HTTP protocol . It is an application layer protocol in the TCP/IP protocol system, an IETF RFC standard submitted by Columbia University, Netscape and RealNetworks.
This protocol defines how one-to-many applications can effectively transmit multimedia data through IP networks. RTSP is architecturally located on top of RTP and RTCP, and it uses TCP or RTP to complete data transmission . RTSP is used to establish control media streams Transport, which acts as a "network remote control" for multimedia services. Although sometimes RTSP control information and media data streams can be interleaved for transmission, in general RTSP itself is not used to transfer media stream data. The transmission of media data can be accomplished through protocols such as RTP/RTCP.
This protocol is used in the C/S model and is a text-based protocol for establishing and negotiating a real-time streaming session between the client and server.
network system
RTSP is an application layer protocol similar to http. A typical streaming media framework network system can refer to the following figure
Explanation of key concepts
- RTSP (Real Time Streaming Protocol), RFC2326, real-time streaming protocol, is an application layer protocol in the TCP/IP protocol system, an IETF RFC standard submitted by Columbia University, Netscape and RealNetworks. The protocol defines how one-to-many applications can efficiently transmit multimedia data over IP networks. RTSP is architecturally located on top of RTP and RTCP, and it uses TCP or UDP to complete data transmission.
- Real-time Transport Protocol or RTP for short, it was published in RFC 1889 in 1996 by the multimedia transmission working group of IETF. The RTP protocol specifies a standard packet format for delivering audio and video over the Internet. It is created on UDP protocol.
- Real-time Transport Control Protocol or RTP Control Protocol or RTCP for short) is a sister protocol of Real-time Transport Protocol (RTP). RTCP is defined by RFC 3550 (replacing obsolete RFC 1889). RTP uses an even UDP port; while RTCP uses the next port of RTP, which is an odd port. RTCP works in tandem with RTP, with RTP doing the actual data transfer and RTCP delivering the control packets to everyone on the phone. Its main function is to give feedback on the quality of service that RTP is providing.
- RTSP protocol: responsible for the request and response between the server and the client
- RTP protocol: responsible for transmitting media data
- RTCP protocol: Provide transmission information during RTP transmission
rtsp is carried on top of rtp and rtcp, rtsp does not send media data, but uses rtp protocol to transmit
rtp does not specify the sending method, you can choose udp sending or tcp sending
Detailed explanation of rtp protocol
background knowledge
Streaming is a new concept that has appeared on the Internet in recent years. Its definition is very broad, and it mainly refers to the general term for the technology of transmitting multimedia data through the network.
There are two types of streaming
- Progressive Streaming
- Real time streaming
Live streaming is real-time delivery, especially for live events. "Real time" means that in an application, the delivery of data must maintain a precise time relationship with the generation of data, which requires corresponding protocol support, so that RTP and RTCP appear accordingly
Principle of rtp protocol
Relatively simple, it is responsible for packaging streaming media data and realizing real-time transmission of media streams, that is, it encapsulates streaming media data according to the RPT data packet format, and uses the protocol bound to it to transmit data packets.
RTP chooses an unused even UDP port number between port numbers 1025 and 65535, while RTCP in the same session uses the next base UDP port number. The RTP default port number is 5004, so the RTCP port number is 5005 by default.
It can be seen from the figure below that RTP is divided into the transport layer, which is built on UDP. Like the UDP protocol, in order to realize its real-time transmission function, RTP also has a fixed encapsulation form. RTP is used to provide time information and stream synchronization for end-to-end real-time transmission, but does not guarantee the quality of service. Quality of service is provided by RTCP.
rtp protocol encapsulation
Detailed explanation:
- Padding bit (1bit) If p=1, one or more extra octets are filled at the end of the message, which are not part of the payload. Padding may be used in certain fixed-length encryption algorithms or to transmit multiple RTP packets in underlying data units
- Extension (X): 1 bit, set to "1" means that the RTP header is followed by an extension header
- Number of participating sources (CSRC count (CC)) 4 bits, CSRC count includes the number of CSRC identifiers immediately after the fixed header.
- Flag (M): 1 bit, its specific interpretation is defined by the application documentation. For example, for a video stream, it indicates the end of a frame, and for audio, the beginning of a conversation
- Payload type, 7 bits, which indicates the load type of the data carried in the user data field, which encoding is used for the data behind the record, and the receiving end finds out the corresponding decoder to decode it
- The sequence number is 16 bits. Every time an RTP data packet is sent, the sequence number is increased by one, and the receiver can detect packet loss and reconstruct the packet sequence accordingly. The initial value of the sequence number is random (unpredictable), so that even if the source itself is not encrypted (sometimes the packet goes through a translator, which it does), plain text attacks on well-known encryption algorithms are also more difficult
- Timestamp, 32 bits, the time scale reflects the sampling instant of the first octet in the RTP packet. The sampling instant must be derived from a monotonically, linearly increasing clock to allow synchronization and jitter calculations. The time stamp can let the receiver know to play the data at the correct time. There is only serial number but no time stamp, and the data cannot be played out completely in order, because if there is no data in the middle of the data, only the serial number will cause an error.
- SSRC, 32 bits, the SSRC segment identifies the synchronization source. This identifier is not randomly selected, and the purpose is to ensure that no two synchronization sources in the same RTP packet connection have the same SSRC identifier. Although the probability of multiple sources choosing the same identity is low, all RTP implementations MUST detect and resolve conflicts.
- CSRC list, 0 to 15 items, each 32 bits. The CSRC list indicates the sources within the packet that contribute to the payload. The identification number is given by the CC segment. If there are more than 15 sources of action, only 15 will be identified. .
Detailed explanation of rtcp protocol
RTCP is also transmitted by UDP, but RTCP encapsulates only some control information, so the packet is very short, so multiple RTCP packets can be encapsulated in one UDP packet. RTCP has the following five packet types.
Detailed explanation of rtsp protocol
RTSP (Real-Time Stream Protocol) protocol is a text-based multimedia playback control protocol, which belongs to the application layer . RTSP works in the client mode, and provides operations such as playing, pausing, rewinding, and forwarding for streaming media. This standard is specified by IETF, and the corresponding protocol is RFC2326. As an application layer protocol, RTSP provides an extensible framework, which makes the control and on-demand of streaming media possible. It is mainly used to control the transmission of data with real-time characteristics, but it is not used to transmit streaming media itself. However, it must rely on the services provided by the underlying transport protocol (such as RTP/RTCP) to complete the transmission of streaming media data. RTSP is responsible for defining specific control information, operation methods, status codes, and describing the interaction with RTP . The RTSP media service protocol framework is as follows:
Therefore, it can be seen from the above architecture diagram that RTSP is used in conjunction with RTP and RTCP. RTSP generally transmits streams in TS and MP4 formats, and its transmission generally requires 2 to 3 channels, and the command and data channels are separated. Using the RTSP protocol to transmit streaming media data requires a dedicated media player and media server, that is, a client and server that support the RTSP protocol.
A basic RTSP operation process:
- First, the client connects to the streaming server and sends an RTSP describe command (DESCRIBE).
- The stream server provides feedback through an SDP description, and the feedback information includes information such as stream quantity and media type.
- The client analyzes the SDP description again, and sends an RTSP setup command (SETUP) for each stream in the session, and the RTSP setup command tells the server client the port used to receive media data. After the streaming media connection is established,
- The client sends a play command (PLAY), and the server starts to transmit the media stream (RTP packet) to the client on UDP. During playback, the client can also send commands to the server to control fast forward, rewind and pause.
- Finally, the client can send a termination command (TERADOWN) to end the streaming session
sequenceDiagram:
客户端->>服务器:DESCRIBE
服务器->>客户端: 200 OK (SDP)
客户端->>服务器:SETUP
服务器->>客户端: 200 OK
客户端->>服务器:PLAY
服务器->>客户端: (RTP包)
Protocol Features
- Extensibility: New methods and parameters are easily added to RTSP.
- Easy to parse: RTSP can be parsed by standard HTTP or MIME parsers.
- Security: RTSP uses web security mechanisms.
- Independent of transmission: RTSP can use Unreliable Datagram Protocol (EDP), Reliable Datagram Protocol (RDP); if you want to achieve application-level reliability, you can use Reliable Stream Protocol.
- Multi-server support: each stream can be placed on different servers, the client automatically establishes several concurrent control connections with different servers, and media synchronization is performed at the transport layer.
- Recording Device Control: Protocol to control recording and playback devices.
- Separation of flow control and meeting start: only required for the meeting initiation protocol, or can be used to create a unique meeting identification number. In special cases, SIP or H.323 can be used to invite the server to join the meeting.
- Suitable for professional applications: RTSP supports frame-level precision through SMPTE time stamping, allowing remote digital editing.
- Demo Descriptions are neutral: the protocol does not impose a specific demo or metafile, and the type of format used may be delivered; however, Demo Descriptions MUST at least include an RTSP URL.
- Proxy and firewall friendly: protocol can be handled by application and transport layer firewalls. Firewalls need to understand the SETUP method, which opens a "gap" for UDP media streams.
- HTTP Friendly: Here, RTSP wisely adopts the HTTP concept, making existing structures reusable. Structures include Platform for Internet Content Selection (PICS). Since controlling continuous media requires server state in most cases, RTSP does not just add methods to HTFP .
- Appropriate server control: If a user starts a stream, it MUST and MAY also stop a stream.
- Transport Coordination: Before actually processing the continuous media stream, the user can coordinate the transport method.
- Performance coordination: If the basic feature is invalid, there must be some cleanup mechanism for the user to decide which method is not effective. This allows the user to propose a suitable user interface.
The difference between RTSP protocol and HTTP protocol
- RTSP introduces several new methods, such as DESCRIBE, PLAY, SETUP, etc., and has different protocol identifiers, RTSP is rtsp 1.0, HTTP is http 1.1;
- HTTP is a stateless protocol, while RTSP maintains state for each session;
- Both the client and the server of the RTSP protocol can send Request requests, while in the HTTP protocol, only the client can send Request requests.
- In the RTSP protocol, the payload data is generally transmitted in an out-of-band manner (except for the case of interleaving), and the payload data is transmitted in different channels through the RTP protocol. The payload data of the HTTP protocol is transmitted in-band, for example, the requested webpage data is carried in the response message body.
- Use ISO 10646 (UTF-8) instead of ISO 8859-1 to match current HTML internationalization;
- RTSP includes absolute URIs when using URI requests. Due to backward compatibility issues caused by historical reasons, HTTP/1.1 only includes absolute paths in the request, and puts the host name in a separate header field;
rtsp message protocol
rtsp data format
RTSP protocol format is similar to HTTP protocol format
RTSP client request format
method url vesion\r\n
CSeq: x\r\n
xxx\r\n
...
\r\n
- method: method, indicating the method of this request, rtsp defines many methods, which will be introduced later
- url: The format is generally rtsp://ip:port/session, ip represents the host ip, port represents the port, if not written, then it is the default port, the default port of rtsp is 554, and session indicates which session is requested
- version: Indicates the version of rtsp, which is now RTSP/1.0
- CSeq: serial number, each RTSP request and response corresponds to a serial number, and the serial number is incremented
The response format of the RTSP server
vesion 200 OK\r\n
CSeq: x\r\n
xxx\r\n
...
\r\n
- version: Indicates the version of rtsp, which is now RTSP/1.0
- CSeq: serial number, this must be the same as the serial number of the corresponding request
Syntax structure of RTSP URL
An end user is the first step to start watching the streaming media service by entering the URL address in the player. For mobile streaming media on demand using the RTSP protocol, the URL is generally written as follows:
A URL link starting with "rtsp" or "rtspu" is used to specify the current RTSP protocol. The syntax structure of RTSP URL is as follows:
rtsp_url = (”rtsp:” | ”rtspu:” | ”rtsps:”) “//” host [“:”port”] /[abs_path]/content_name
- rtsp: Use a trusted underlying transport protocol, such as TCP
- rtspu: Use an untrusted underlying transport protocol, such as UDP
- rtsps: Use a trusted encrypted transport protocol, such as TCP + TLS
- host: Can be a valid domain name or IP address.
- port: port number, for RTSP protocol, the default port number is 554 . When we confirm that the port number provided by the streaming media server is 554, this item can be omitted: when the port number used by the HMS server is 554, we do not need to specify the port number when writing an on-demand link, but when using a non- When using port 554, the corresponding port must be specified in the RTSP URL.
- abs_path: identifies the media stream resource in RTSPServer
- RTSPURL is used to identify the media stream resource of RTSPServer, which can identify a single media stream resource or a collection of multiple media stream resources.
For example, a full RTSP URL can be written as:
rtsp://192.168.1.67:554/test
Another example is the RTSP address format of the commonly used Haikang network camera on the market:
rtsp://[username]:[password]@[ip]:[port]/[codec]/[channel]/[subtype]/av_stream
example
rtsp://admin:[email protected]:554/h264/ch1/main/av_stream
Another simple example is as follows:
rtsp://media.example.com:554/twister/audiotrack
Let's take a look at the abs path = twister/audiotrack for the URL above. The twister represents a presentation, and the presentation consists of one or more real-time streams. audiotrack represents the name of one of the real-time streams in the identification (Presentation). As can be seen from the name, what we want to fetch is an audio stream. If abs path = twister/videotrack, it means that we want to get the video stream of twister.
Some servers also support the following URL forms:
rtsp://media.example.com:554/twister
The URL indicates the video stream and audio stream of the presentation (Presentation).
RTSP message structure
RTSP is a text-based protocol that uses CRLF as the end of a line. The advantage of using a text-based protocol is that we can add custom parameters at any time during use, and we can also grab the protocol package at will and analyze it intuitively.
RTSP has two types of messages: request messages and response messages. The request message refers to the request message sent from the client to the server, and the response message refers to the answer from the server to the client. Since RTSP is text-oriented, each field in the message is some ASCII code string, so the length of each field is uncertain. The RTSP message consists of three parts, namely the start line, the header line and the entity body. In a request message, the start line is the request line.
RTSP request message
The methods of RTSP request message include: OPTIONS, DESCRIBE, SETUP, TEARDOWN, PLAY, PAUSE, GET_PARAMETER and SET_PARAMETER.
A request message (a request message) can be initiated from the client to the server or from the server to the client. The syntax structure of the request message is as follows:
Request = Request-Line
*( general-header | request-header | entity-header)
CRLF
[message-body]
Request Line
The syntax structure of the first line of the request message is as follows:
Request-Line = Method 空格 Request-URI 空格 RTSP-Version CRLF
where the first word that appears in the message line is the signaling flag used. Currently available information flags are as follows:
Method = “DESCRIBE”
| “ANNOUNCE”
| “GET_PARAMETER”
| “OPTIONS”
| “PAUSE”
| “PLAY”
| “RECORD”
| “REDIRECT”
| “SETUP”
| “SET_PARAMETER”
| “TEARDOWN”
example:
DESCRIBE rtsp://211.94.164.227/3.3gp RTSP/1.0
Request Header Fields
In addition to the content of the first line in the message header, there are some requirements to provide additional information. Some of them are necessary, and we will introduce the meaning of several frequently used fields in detail later.
Request-header = Accept
| Accept-Encoding
| Accept-Language
| Authorization
| From
| If-Modified-Since
| Range
| Referer
| User-Agent
response message
The start line of the response message is the status line, and the structure of the RTSP response message is shown in the figure below
The syntax structure of the response message is as follows:
Response = Status-Line *( general-header | response-header | entity-header) CRLF [message-body]
Status-Line
The first line of the response message is the status-line, with each element separated by a space. Except for the final CRLF, there must be no occurrences of CR or LF in the middle of the line. Its syntax is as follows,
Status-Line = RTSP-Version 空格Status-Code 空格Reason-Phrase CRLF
The status code (Status-Code) is a three-digit integer used to describe the execution result of the received request message by the receiver
The first digit of the Status-Code specifies the type of the reply message, and there are 5 types in total:
- 1XX: Informational – request received, continue processing
- 2XX: Success – the request was successfully received, parsed and accepted
- 3XX: Redirection – more action is required to complete the request
- 4XX: Client Error – The request message contains syntax errors or cannot be executed effectively
- 5XX: Server Error – The server failed to respond and could not process a valid request message correctly
Response Header Fields
What is stored in the field of the response message is some additional information that cannot be placed in the Status-Line and needs to be sent to the requester.
Response-header = Location
| Proxy-Authenticate
| Public
| Retry-After
| Server
| Vary
| WWW-Authenticate
The main method of RTSP
Note: P—presentation, C—client, S—server, S (object column)—stream
RTSP important header field parameters
- Accept: Used to specify the type of media description information that the client can accept. For example: Accept: application/rtsl, application/sdp;level=2
- Bandwidth: Used to describe the bandwidth value available to the client.
- CSeq: Specifies the serial number of the RTSP request response pair, which must be included in each request or response. For every request message containing a given sequence number, there will be a response message with the same sequence number.
- Rang: used to specify a time range, you can use SMPTE, NTP or clock time unit.
- Session: The Session header field identifies an RTSP session. The Session ID is selected by the server in the SETUP response. Once the client obtains the Session ID, it will include the Session ID in subsequent operation request messages for the Session.
- Transport: The Transport header field contains a list of transport options acceptable to the client, including transport protocol, address port, TTL, etc. The server side also returns the specific option actually selected through this header field. Such as: Transport: RTP/AVP;multicast;ttl=127;mode="PLAY", RTP/AVP;unicast;client_port=3456-3457;mode="PLAY"
Simple RTSP message interaction process
C means RTSP client, S means RTSP server
Step 1: Query the methods available on the server side
C->S OPTION request //询问S有哪些方法可用
S->C OPTION response //S回应信息的public头字段中包括提供的所有可用方法
Step 2: Get media description information
C->S DESCRIBE request //要求得到S提供的媒体描述信息
S->C DESCRIBE response //S回应媒体描述信息，一般是sdp信息
Step 3: Establish an RTSP session
C->S SETUP request //通过Transport头字段列出可接受的传输选项，请求S建立会话
S->C SETUP response //S建立会话，通过Transport头字段返回选择的具体转输选项，并返回建立的Session ID;
Step 4: Request to start transferring data
C->S PLAY request //C requests S to start sending data
S->C PLAY response //S responds to the requested information
Step 5: Data transmission is playing
S->C Send streaming data//Transmit data through RTP protocol
Step 6: Close the session and exit
C->S EARDOWN request //C requests to close the session
S->C TEARDOWN response //S responds to the request
The above process is just a standard and friendly rtsp process, but it does not necessarily follow this process in actual needs. The third and fourth steps are required! In the first step, as long as the server and the client agree on which methods are available, the option request can be omitted. In the second step, if we have other ways to get the media initialization description information (such as http requests, etc.), we don't need to complete it through the describe request in rtsp.
RTSP request response example
Where C is the client and S is the server.
OPTIONS
- C–>S
OPTIONS rtsp://192.168.31.115:8554/live RTSP/1.0\r\n
CSeq: 2\r\n
\r\n
The client requests available methods from the server
- S–>C
RTSP/1.0 200 OK\r\n
CSeq: 2\r\n
Public: OPTIONS, DESCRIBE, SETUP, TEARDOWN, PLAY\r\n
\r\n
The server replies to the client, currently available methods OPTIONS, DESCRIBE, SETUP, TEARDOWN, PLAY
DESCRIBE
- C–>S
DESCRIBE rtsp://192.168.31.115:8554/live RTSP/1.0\r\n
CSeq: 3\r\n
Accept: application/sdp\r\n
\r\n
The client requests a media description file from the server in the format of sdp
- S–>C
RTSP/1.0 200 OK\r\n
CSeq: 3\r\n
Content-length: 146\r\n
Content-type: application/sdp\r\n
\r\n
v=0\r\n
o=- 91565340853 1 in IP4 192.168.31.115\r\n
t=0 0\r\n
a=contol:*\r\n
m=video 0 RTP/AVP 96\r\n
a=rtpmap:96 H264/90000\r\n
a=framerate:25\r\n
a=control:track0\r\n
The server replied with the sdp file, which tells the client what audio and video streams the current server has, and what attributes they have, which will be explained later
Here it is only necessary until the client can know which audio and video streams can be sent based on this information
SETUP
- C–>S
SETUP rtsp://192.168.31.115:8554/live/track0 RTSP/1.0\r\n
CSeq: 4\r\n
Transport: RTP/AVP;unicast;client_port=54492-54493\r\n
\r\n
The client sends an establishment request, requests to establish a connection session, and is ready to receive audio and video data
Analyze Transport: RTP/AVP;unicast;client_port=54492-54493\r\n
- RTP/AVP: indicates that RTP is sent via UDP, if it is RTP/AVP/TCP, it indicates that RTP is sent via TCP
- unicast: means unicast, if it is multicast, it means multicast
- client_port=54492-54493: Since RTP OVER UDP is desired here, the client sends two ports for data transmission, and the client has bound these two ports to two udp sockets, 54492 It means RTP port, 54493 means RTCP port (RTP port is an even number, RTCP port is RTP port + 1)
2.S–>C
RTSP/1.0 200 OK\r\n
CSeq: 4\r\n
Transport: RTP/AVP;unicast;client_port=54492-54493;server_port=56400-56401\r\n
Session: 66334873\r\n
\r\n
- After the server receives the request, it knows that the client requires RTP OVER UDP to send data, unicast, the port used by the client to transmit RTP data is 54492, and the port for RTCP is 54493
- The server also has two udp sockets, two ports are bound, one is used to transmit RTP, and the other is used to transmit RTCP. The port number here is 56400-56401
- After that, the client will use the ports 54492-54493 to transmit data with the server through udp, and the server will use the ports 56400-56401 to transmit data with the client
PLAY
- C–>S
PLAY rtsp://192.168.31.115:8554/live RTSP/1.0\r\n
CSeq: 5\r\n
Session: 66334873\r\n
Range: npt=0.000-\r\n
\r\n
Client requests to play media
- S–>C
RTSP/1.0 200 OK\r\n
CSeq: 5\r\n
Range: npt=0.000-\r\n
Session: 66334873; timeout=60\r\n
\r\n
After the server replies, it will start using RTP to send data to the client's port 54492 through udp
TEARDOWN
- C–>S
TEARDOWN rtsp://192.168.31.115:8554/live RTSP/1.0\r\n
CSeq: 6\r\n
Session: 66334873\r\n
\r\n
- S–>C
RTSP/1.0 200 OK\r\n
CSeq: 6\r\n
\r\n
Reposted from: https://zhuanlan.zhihu.com/p/478736595