1. The basic process of RTMP transmission
RTMP video and audio data flow
sender
-
Step 1: Encapsulate the data into a message (Message).
-
Step 2: Divide the message into message blocks (Chunk, the content actually transmitted in the network).
-
Step 3: Send the divided message block (Chunk) through the TCP protocol.
Receiving end:
-
Step 1: After receiving the data through the TCP protocol, first reassemble the message block into a message (Message).
-
Step 2: The data can be recovered by decapsulating the message.
The outstanding advantages of RTMP are: low delay , the normal delay is 1~3 seconds, ideally, the live broadcast can be controlled within 1 second. How did it do it? What are some good design ideas?
2. RTMP design idea
RTMP design ideas
2.1 Big and small, light and simple
The basic data unit in the RTMP protocol is called a message (Message). When transmitting data on the Internet, the message (Message) will be split into smaller units, called the message block (Chunk). The large Message is cut into small Chunk that is convenient for transmission on the network . I personally think that this is one of the RTMP So the core reason for the low latency .
Message is split into Chunk
Cutting into small blocks also prevents large data blocks (such as video data) from blocking small data blocks (such as audio data or control information) .
2.2 This is a fellow traveler from the end of the world
RTMP is based on the TCP protocol, including control information, video data, and audio data all share the same TCP Connection. That is to say, there is only one way. And only one car is allowed to pass at a time. Generate good video chunks and audio chunks and go on the road in turn .
Schematic diagram of the transmission sequence of Video Chunk and Audio Chunk on the network
This involves a very important concept in RTMP: multiplexing . The beginning of the RTMP protocol is introduced as follows:
Abstract This memo describes Adobe’s Real Time Messaging Protocol (RTMP), an application-level protocol designed for multiplexing and packetizing multimedia transport streams (such as audio, video, and interactive content) over a suitable transport protocol (such as TCP).
Among them , multiplexing means multiplexing , which refers to the process of integrating independent audio and video data into a data stream so that the audio and video streams can be transmitted synchronously . In RTMP live broadcast, video chunks and audio chunks are generated in real time, added to the data stream in turn, and sent to the client through the network. Multiplexing is the transmission method in RTMP and the key to audio and video synchronization .
[Learning address]: FFmpeg/WebRTC/RTMP/NDK/Android audio and video streaming media advanced development
[Article Benefits]: Receive more audio and video learning packages, Dachang interview questions, technical videos and learning roadmaps for free. The materials include (C/C++, Linux, FFmpeg webRTC rtmp hls rtsp ffplay srs, etc.) Click 1079654574 to join the group to receive it~
2.3 Matters are prioritized, military orders are the most important, and food and grass are the second
In RTMP, messages (Message) are mainly divided into two categories: control messages and data messages. Data messages include Video messages and Audio messages. How are these messages managed? It's a bit like the management of vehicles. That is to say, there is only one road. Who will go first and who will go later? The answer is: priority is given, and the one with the highest priority goes first . A lower priority cannot block a higher priority.
The RTMP protocol can be divided into two levels as a whole:
RTMP protocol layer
Message streamIt belongs to the application level message and Chunk streambelongs to the lower layer of RTMP protocol.
RTMP Chunk Stream does not provide any prioritization or similar forms of control, but can be used by higher-level protocols to provide such prioritization. For example, a live video server might choose to drop video messages for a slow client to ensure that audio messages are received in a timely fashion, based on either the time to send or the time to acknowledge each message.
RTMP Chunk StreamHierarchy does not have prioritization, but Message streamprovides prioritization at a high level . That is to say: different types of messages will be assigned different priorities . When the network transmission capacity is limited, the priority is used to control the queuing order of messages at the bottom layer of the network. For example, when the client network is poor, the streaming media server may choose to discard the video message to ensure that the audio message can be delivered to the client in time.
Chunking allows large messages at the higher-level protocol to be broken into smaller messages, for example to prevent large low priority messages (such as video) from blocking smaller high-priority messages (such as audio or control).
RTMP Chunk StreamHierarchy allows Message streamlarge messages to be cut into smaller ones at a level, so that large low-priority messages (such as video messages) can be prevented from blocking small high-priority messages (such as audio messages or control messages).
RTMP message priority indication
Protocol Control Messages RTMP Chunk Stream uses message type IDs 1, 2, 3, 5, and 6 for protocol control messages. These messages contain information needed by the RTMP Chunk Stream protocol. These protocol control messages MUST have message stream ID 0 (known as the control stream) and be sent in chunk stream ID 2. Protocol control messages take effect as soon as they are received; their timestamps are ignored.
Protocol Control MessagesControl messages belonging to RTMP Chunk Streamthe hierarchy are used for internal control of the protocol.
User Control Messages (4) RTMP uses message type ID 4 for User Control messages. These messages contain information used by the RTMP streaming layer. Protocol messages with IDs 1, 2, 3, 5, and 6 are used by the RTMP Chunk Stream protocol (Section 5.4). User Control messages SHOULD use message stream ID 0 (known as the control stream) and, when sent over RTMP Chunk Stream, be sent on chunk stream ID 2. User Control messages are effective at the point they are received in the stream; their timestamps are ignored.
User Control Messages (4)Yes RTMP streaming layer(that is, the Message stream level) message.
-
The most important protocol control messages ( Protocol Control Messages ) and user control messages ( User Control Messages ) should contain message flow ID 0 (control flow) and block flow ID 2, and have the highest sending priority .
-
Secondary data messages (audio messages, audio messages) have lower priority than control messages. Also, in general, audio messages have higher priority than video data.
2.4 Everything is negotiable
The sender, in the process of cutting the Message into Chunks, uses the Chunk Size (the default value is 128 bytes) as the basis for cutting.
Message is divided into Chunk example
Larger chunk sizes reduce CPU usage, but also commit to larger writes that can delay other content on lower bandwidth connections. Smaller chunks are not good for high bit rate streaming.
The larger the Chunk Size, the smaller the burden on the CPU during cutting; however, in an environment where the bandwidth is not sufficient, sending is time-consuming and will block the sending of other messages. The smaller the Chunk Size is, the better it is for network sending, but the CPU burden on the server is heavy. It is not suitable for high bit rate data streams.
The Chunk Size can be changed according to the actual situation, that is, it Set Chunk Sizecan be updated by sending a control command ( ).
The chunk size is configurable. It can be set using a Set Chunk Size control message (Section 5.4.1). Chunk size is maintained independently for each direction. Protocol control message 1, Set Chunk Size, is used to notify the peer of a new maximum chunk size. The maximum chunk size defaults to 128 bytes. The maximum chunk size SHOULD be at least 128 bytes, and MUST be at least 1 byte.
Fully consider the situation of the streaming media server, bandwidth, and client, and Set Chunk Sizeachieve the optimal effect by dynamically adapting to the environment.
2.5 Lazy when you can
RTMP Chunk HeaderThe length is not fixed, divided into: 12 Bytes、8 Bytes、4 Bytes、1 Bytefour.
RTMP Chunk Header (12 Bytes)
Chunking also allows small messages to be sent with less overhead, as the chunk header contains a compressed representation of information that would otherwise have to be included in the message itself.
Under normal circumstances, msg stream idit will not change, so for video or audio, except for the first RTMP Chunk Header which is 12Bytes, the follow-up can use 8Bytes.
RTMP Chunk Header (8 Bytes)
If the length ( message length) and type ( msg type id, such as video 9or audio 8) of the message are the same, these two parts can also be omitted, and RTMP Chunk Headerthe 4Bytes type is used.
RTMP Chunk Header (4 Bytes)
If the current ChunkChunk is the same as the previous Chunk, msg stream idthe msg type id is the same, the message length is the same, and they all belong to the same message (cut from the same Message), the timestamp of this type of Chunk is also the same, so the subsequent can also be omitted, RTMP Chunk Headerusing 1 Byte type.
RTMP Chunk Header (1 Byte)
When Chunk Sizeit is large enough (generally not done), all of Messagethem can only be cut into one correspondingly at this time Chunk, which should Chunkonly msg stream idbe the same. At this time, basically except for the first one Chunk, everything else is .Header12BytesChunkHeader8Bytes
Chunk Header situation when the Chunk Size is large enough
Hong Kong Satellite TV's RTMP stream: rtmp://live.hkstv.hk.lxdns.com/live/hksThis is the case.
The size of the server Set Chunk Size
It can be seen that the Chunk Size is very large, for 2147483650.
3. Summary
The core idea of RTMP design: block multiplexing and classification There are also some design ideas: the Chunk Size can be flexibly set and the RTMP Chunk Header can be flexibly selected (to achieve the effect of compression).
4、References:
rtmp_specification_1.0.pdf Love ZY Forever
Original text link: RTMP protocol 03 RTMP design idea - short book