1. Introduction to RTMP
RTMP (Real Time Messaging Protocol) is an application layer protocol, mainly used to transmit video, audio, control commands and other content between Flash player and server. The outstanding advantages of this protocol are: low latency.
RTMP is based on TCP and uses port 1935 by default.
2. RTMP noun analysis
Payload: Data contained in a packet, such as audio samples or compressed video data.
Packet: A packet consists of a fixed header and payload data. Some underlying layer protocols may require custom encapsulation of data packets.
Port: An abstraction used by a transport protocol to distinguish between different destinations specifying a host. TCP/IP uses small positive integers to identify ports. The transport selector (TSEL) used by the OSI transport layer is equivalent to a port.
Transport address: A combination of network address and port used to identify a transport layer endpoint, such as an IP address and a TCP port. Data packets are sent from a source transport address to a destination transport address.
Message stream: A logical channel through which messages flow in communication.
Message stream ID (message stream ID): Each message has an associated ID, which can be used to identify the message stream in circulation.
Chunk: A segment of a message. Messages are split into many small parts before being sent over the network. Chunks can ensure end-to-end delivery of all messages with ordered timestamps, even with many different streams.
Chunk stream: A logical channel in communication that allows chunks to flow in a specific direction. Block streams can flow from client to server, or from server to client.
Chunk stream ID (block stream ID): Each block has an associated ID, which can be used to identify the block stream in circulation.
Multiplexing (synthesis): The processing of combining independent audio/video data into a continuous audio/video stream, so that several videos and audios can be sent at the same time.
DeMultiplexing (decomposition): The reverse processing of Multiplexing, which restores the interleaved audio and video data to the format of the original audio and video data.
Remote Procedure Call (RPC remote method call): Allows the client or server to call a subroutine or program request on the opposite end.
Metadata: A description of the data. A movie's metadata includes movie title, duration, creation time, etc.
Application Instance (application instance): The instance of the application on the server, the client can connect to this instance and send a connection request.
Action Message Format (AMF Action Message Format Protocol): A compact binary format for serializing ActionScript object graphs. There are two versions of AMF: AMF O [AMFO] and AMF 3 [AMF3].
3. RTMP push and pull stream
3.1 FFMPEG streaming+FFPLAY playback
推流：ffmpeg -re -i /mnt/hgfs/linux/vod/35.mp4 -c copy -f flv rtmp://192.168.100.41/live/35
拉流：ffplay rtmp://192.168.100.41/live/353.2 FFPLAY playback
拉流：ffplay rtmp://202.69.69.180:443/webcast/bshdlive-pc4. Basic process of RTMP playback
4.1 Push flow process
4.2 Streaming process
4.3 Step 1: TCP three-way handshake - repairing the highway
RTMP is an application layer protocol based on TCP. Through the TCP three-way handshake, the RTMP client can establish a reliable network connection with the designated port of the RTMP server (the default port is 1935). The network connection here is the real physical connection. After completing the three-way handshake, the client and server can start transmitting data.
[Image upload failed...(image-46c148-1681736408459)]
image.png
After three handshakes, the client and the server establish a TCP Connection on port 1935.
[Learning address]: FFmpeg/WebRTC/RTMP/NDK/Android audio and video streaming media advanced development
[Article Benefits]: Receive more audio and video learning packages, Dachang interview questions, technical videos and learning roadmaps for free. The materials include (C/C++, Linux, FFmpeg webRTC rtmp hls rtsp ffplay srs, etc.) Click 1079654574 to join the group to receive it~
4.4 Step 2: RTMP handshake - security check
Instead of calling it an RTMP handshake, it actually plays the role of verification.
The basic process of RTMP handshake:
RTMP handshakes are mainly divided into: simple handshake and complex handshake.
4.5 Step 3: RTMP handshake - simple handshake
Protocol version number (8bit) C0: Client version S0: Server version is currently 3, (0, 1, 2 has been discarded) The length of both C1 and S1 data packets is 1536 bytes
03(C0) 00(C1 start) 00 00 00 09 00 7c 02 f7 78 55 1e ce ab 8e S1 segment: 00 90 65 30 0d 0e 0a 0d 64 84 1c ad 1e 7f 0c C2 and S2 packet lengths are both 1536 Byte, basically a copy of S1 and C1.
S2 fragment
4.6 Step 4: RTMP handshake – complex handshake
Compared with the simple handshake, the complex handshake mainly adds stricter verification. Mainly divide the 1528Bytes random number in the simple handshake into two parts, one part of 764Bytes stores the public key (public key), and the other part of 764Bytes stores the digest (ciphertext, 32 bytes).
In addition, another obvious feature of the complex handshake is that the Version part is not 0, and the server can judge whether it is a simple handshake or a complex handshake based on this.
4.7 Step 5: connect
This is also called connection, what is the connection? Here you must understand a very important concept in RTMP: Application Instance.
Different Application Instances can be distinguished according to their functions. For example, live broadcast can be represented by live, and playback on demand can be represented by vod.
rtmp://192.168.100.41/live/36 where live is Application Instance(sport, music) When playing the stream, the address of connect is rtmp://192.168.100.41/live/36
4.8 Step 6: createStream (create stream) - create a logical channel
The createStream command is used to create a logical channel, which is used to transmit video, audio, and metadata.
It will be returned in the server's response message to uniquely identify the Stream.
Note: The difference between Message ID and Stream ID
The command structure from the client to the server is as follows:
The command structure from server to client is as follows:
4.9 Step 7: play
The client sends the play command to play the specified stream. Start transmitting audio and video data. If you want to play immediately after sending the play command, you need to clear other streams in the play queue and set reset to true.
4.10 Step 8: deleteStream (delete stream)
The command structure from the client to the server is as follows:
Delete the stream with the specified Stream ID. The server does not need to send a response message to this command.
5. RTMP level
RTMP level (data transmission angle) RTMP level (data reception angle) RTMP level (protocol angle)
5.1 RTMP level (data perspective)
5.1.1 RTMP level (data transmission angle)
5.1.2 RTMP level (data receiving angle)
6. RTMP key structure
Message & Chunk
An important concept in Message RTMP is the message.
6.1 Message Classification – Message Type
Messages are mainly divided into three categories: protocol control messages, data messages, command messages, etc.
protocol control message
Message Type ID = 1 2 3 5 6 and Message Type ID = 4 are mainly used for control within the protocol, which will be analyzed in detail later in this part.
data message
Message Type ID = 8 9 18 8: Audio audio data 9: Video video data 18: Metadata Including audio and video encoding, video width and height and other information.
Command Message Command Message (20, 17)
This type of message mainly includes two classes, NetConnection and NetStream, each of which has multiple functions. The call of this message can be understood as a remote function call.
Message Classification – Stream ID
Message StreamID is the unique ID of the audio and video stream. If a stream has both audio and video packets, the StreamID of the audio packet of this stream is the same as the StreamID of its video packet.
6.2 Message & Chunk
What is actually sent in the Chunk network.
image.png
6.2.1 Chunk Stream ID
Each chunk that is created has a unique ID associated with it called chunk stream ID 。(5.3. Chunking)
Question: Since Chunks of multiple message types can be interleaved in a stream, how do multiple Chunks mark messages that belong to the same type of Message?
Answer: It is distinguished by the Chunk Stream ID, and the same Chunk Stream ID must belong to the same Message
Video and audio in RTMP streams have separate Chunk Stream IDs, such as cs id=20 for audio and cs id=21 for video. After receiving the Chunk, the receiving end will "combine the audio and video into a message" according to the cs id.
6.3 Message & Chunk
Message is cut into one or more Chunks, and then sent on the network.
When sending, the next chunk can only be sent after a chunk is sent.
When splitting, the default Chunk Size is 128 bytes. Take the Message size of 300 bytes as an example to split. 300 = 128 + 128 + 44
7. The essence of RTMP
-
The sender first processes the data into a message (intermediate), then divides the message into Chunk (plus Chunk Header), and then sends the Chunk over the network.
-
Receiver The receiver assembles the received Chunk into a message.
7.1 RTMP level (protocol perspective)
7.2 RTMP Chunk Header
The length of RTMP Chunk Header is not fixed. It is divided into four types: 12 Bytes, 8 Bytes, 4 Bytes, and 1 Byte, which are determined by the first two bits of RTMP Chunk Header.
Correspondence between chunk type and chunk header length
7.3 Comparison of 4 types
7.4 RTMP Chunk Header
7.5 RTMP Chunk Header-Why are there different lengths?
Under normal circumstances, the msg stream id will not change, so for video or audio, except for the first RTMP Chunk Header which is 12Bytes, the follow-up can use 8Bytes. (5.3. Chunking)
If the message length (message length) and type (msg type id, such as video is 9 or audio is 8) are the same, these two parts can also be omitted, and the RTMP Chunk Header adopts the 4Bytes type.
If the current Chunk has the same msg stream id, the same msg type id, and the same message length as the previous Chunk, and they all belong to the same message (cut from the same Message), the timestamp of this type of Chunk is also the same Therefore, the subsequent ones can also be omitted, and the RTMP Chunk Header uses 1 Byte type.
When the Chunk Size is large enough (generally not), all Messages can only be cut into one Chunk at this time, and only the msg stream id of the Chunk is the same. At this time, except for the header of the first Chunk, which is 12 Bytes, the headers of all other Chunks are 8 Bytes.
7.6 Example of RTMP Chunk Header (12 Bytes)
RTMP Header (12 Bytes)
Generally, only the metadata at the beginning of the rtmp stream, the video or audio of the absolute timestamp is 12Bytes.
Some control messages are also 12 Bytes, such as connect.
7.7 Example of RTMP Chunk Header (8 Bytes)
7.8 Example of RTMP Chunk Header (4 Bytes)
7.9 Example of RTMP Chunk Header (1 Byte)
8. Basic process of RTMP transmission
-
Step 1 at the sending end: Encapsulate the data into a message (Message). Step 2: Divide the message into message blocks (Chunk, the content actually transmitted in the network). Step 3: Send the divided message block (Chunk) through the TCP protocol.
-
Receiver: Step 1: After receiving the data through the TCP protocol, first reassemble the message block into a message (Message). Step 2: The data can be recovered by decapsulating the message.
8.1 Why does RTMP divide Message into Chunks?
When transmitting data in the Internet, the message (Message) will be split into smaller units, called the message block (Chunk).
Large messages are cut into small chunks that are convenient for transmission on the network
Cutting into small chunks prevents large chunks of data (such as video data) from blocking smaller chunks of data (such as audio data or control information).
If the data volume of a 1080P I frame is 244KBytes, assuming that the bandwidth is 10Mbit/s, the time taken to transmit a frame is: 244 1024 8/(10 1024 1024) = 0.190625 seconds = 190 milliseconds
If you want to transmit 25 frames of I-frame files in real time, that is, the maximum time allowed for each frame transmission is 40 milliseconds, and the required bandwidth is 47.65625 Mbit, which does not include the ACK data in transmission.
8.2 RTMP message priority
In RTMP, messages (Message) are mainly divided into two categories: control messages and data messages.
Data messages include Video messages and Audio messages.
How are messages managed?
There is only one path (RTMP is a single path), who will go first and who will go later?
The answer is: by priority, the one with the highest priority goes first. A lower priority cannot block a higher priority.
There is no priority division at the RTMP Chunk Stream level, but priority division is provided at the high-level Message stream.
Different types of messages will be assigned different priorities. When the network transmission capacity is limited, the priority is used to control the queuing order of messages at the bottom layer of the network.
For example, when the client network is poor, the streaming media server may choose to discard the video message to ensure that the audio message can be delivered to the client in time.
Note: 5. RTMP Chunk Stream
The RTMP Chunk Stream level allows large messages to be cut into small messages at the Message stream level, which can prevent large low-priority messages (such as video messages) from blocking small high-priority messages (such as audio messages or control messages).
Note: 5.3. Chunking
8.2.1 RTMP message priority - protocol control message
Protocol Control Messages are control messages at the RTMP Chunk Stream level and are used for internal control of the protocol.
8.2.2 RTMP message priority - user control message
Protocol Control Messages are control messages at the RTMP Chunk Stream level and are used for internal control of the protocol.
User Control Messages (4) are messages of the RTMP streaming layer (that is, the Message stream level).
8.3 Summary of RTMP message priority
-
Protocol Control Messages (Protocol Control Messages) and User Control Messages (User Control Messages) should contain message flow ID 0 (control flow) and block flow ID 2, and have the highest sending priority.
-
Data followed by data messages (audio messages, audio messages) have lower priority than control messages. Also, in general, audio messages have higher priority than video data.
8.4 RTMP Protocol - Timestamp
basic introduction
• The unit of timestamp in RTMP is milliseconds (ms) • The timestamp is a relative value relative to a certain time point • The length of the timestamp is 32bit. If rollback is not considered, it can represent a maximum of 49 days, 17 hours, 2 minutes and 47.296 seconds • The Timestamp delta unit is also milliseconds, which is an unsigned integer relative to the previous timestamp; it may be 24bit or 32bit
Message timestamp
Timestamp: Four-byte field that contains a timestamp of the message.
The 4 bytes are packed in the big-endian order.
•Time stamp of RTMP Message is 4 bytes •Big endian storage
8.5 Chunk Timestamp
Using wireshark subpackage analysis, it is found that except for the first video timestamp of the chunk video stream (or audio stream) of the rtmp stream, which is an absolute timestamp, the subsequent timestamps are all timestamp delta, that is, the difference between the current timestamp and the previous timestamp difference.
For example, for a video stream with a frame rate of 25 frames per second, the timestamp delta is basically 40ms.
Normally, Chunk's timestamp (including absolute timestamp and Timestamp delta) is 3 bytes.
But when the timestamp value exceeds 0xFFFFFF, enable Extended Timestamp (4 bytes) to represent the timestamp.
Normally -- 3 bytes
The three-byte timestamp may be absolute timestamp or timestamp delta.
When the value of timestamp delta exceeds 16777215 (that is, 0xFFFFFF in hexadecimal), these three bytes must be set to: 0xFFFFFF to mark Extended
Timestamp (4 bytes) will exist, represented by Extended Timestamp.
Original Link: Audio and Video Streaming Media Development [Forty-six] RTMP Streaming Media 3 - Live Streaming - Short Book