Introduction to RTMP

Introduction

The RTMP protocol is Real Time Message Protocol (abbreviation for Real-time Transport Protocol), and an application layer protocol provided by Adobe to solve the problem of boring data stream transmission.

Multiplexing and packetization issues.

1. RTMP is an application layer protocol that uses TCP to ensure reliable transmission
2. After TCP completes the handshake connection establishment, it will also perform some handshake actions of RTMP, and establish RTMP connection
3. Some control information will be transmitted on the Connetion connection, such as setting SetChunkSize, SetACKWindosSize. The CreateStream command creates some stream connections to transmit specific audio and video data and control these information transmission command information
4. RTMP will format the data into Messages when sending, and then divide each Message into Chunks with Message IDs. Each Chunk may be a part of the entire Message. At the receiving end, the chunk will be restored to a Message according to the Message ID, length and data.

handshake

To establish a valid RTMP Connetion connection, you need to shake hands first: first, the client needs to send three chunks of C0, C1, and C2 (in order) to the server, and the server sends three chunks of S0, S1, and S2 (in order) to the client. , and then effective information transmission can be carried out. The RTMP protocol itself does not stipulate the sequential transmission order of the 6 Messages, but the implementation of the RTMP protocol needs to pay attention to the following points.

1. The client needs to wait for receiving S1 before sending C2
2. Client must receive S2 message, then send other data
3. The server must receive the C0 or C1 message, and then the S0, S1 message
4. The server only receives the C2 message, and then sends other data
   

RTMP Chunk Stream

Message refers to the message sent by Chunk that meets the protocol format and can be divided into chunks. The message mainly includes the following parts,

1. Timestamp (time stamp)
   message timestamp (not necessarily guaranteed to be the current time) 4 bytes
2. Length (length)
   refers to the length of Message Payload (message load) audio and video information, 3 bytes
3. TYpeId (type id)
   id of the message type, 1 byte
4. Message Stream ID (message stream ID) is
   the unique identifier in each message stream. When it is divided into chunks and restored to Messag, it is based on this ID to indicate that the same message belongs to the same chunk, and the 4 chunks of the same message bytes and are stored in little-endian numbers.

During the transmission process in RTMP, the Message is split into chunks for transmission. After the transmission of a Chunk is completed, the next chunk must be transmitted, and the MessageID carried in the chunk is used to identify which Message it belongs to.
By splitting, a message with a large amount of data can be divided into small messages, so that the problem of low priority can be avoided. Continuously send data with high blocking priority, such as during video transmission, or include video frames, audio frames, and RTMP control information. If audio data is continuously sent, video data will be blocked.

For a message with a small amount of data, you can compress the information through the fields of the Chunk Header, thereby reducing the amount of information transmission. (The specific compression method will be introduced later)

The default size of the Chunk is 128 bytes. During the transmission process, the maximum value of the Chunk data volume can be set through a control message called Set Chunk Size. The sending end and the receiving end will each maintain a Chunk Size, which can be set separately. To change the maximum size of the Chunk sent by your side.

Larger chunks reduce the time to calculate each chunk and save CPU, but the sending time is long, and in the case of bandwidth, subsequent messages may be blocked. If the Chunk is too small, additional information headers will be reflected. A small number of multiple times cannot make full use of the high bandwidth, and it is not suitable for transmission in high-bit streams. The actual chunk size should be determined according to the actual situation.

Network connection establishment (NetConnection):

• The client sends a command message to connect to the server, requesting to establish a connection with the server instance
• After receiving the connection command message, the server sends a confirmation window size protocol message to the client, and simultaneously connects to the application program mentioned in the connection fortune
• The server sends a set bandwidth() protocol message to the client
• After the client processes the setting bandwidth protocol message, it sends a confirmation window size protocol message to the server
• The server sends the stream start message in the user control message to the client
• The server notifies the client of the connection status by sending you the result in the command message.
  
  Create a network stream:
• The client sends the create stream command in the message to the server
• After receiving the command stream message from the client, the server sends the result in the command message to notify the client of the stream status.
  
  Play (play):
• The client sends the command message to play the command channel to the server
• After receiving the dial command, the server sends a message protocol to set the size of the block
• The server sends the streambegin in the user control message, telling the client the ID of the stream
• After the play command is successful, the server sends the response status in the command message
• After this send the data that the client needs to send

basic structure

The real content of RTMP, in addition to the handshake, is the message of typeid, which is easier to stream.
The Messa sent in RTMP data is shown in the figure above, and its basic structure:

• header: The header part is used to represent different typeIDs, tell the client the corresponding data type, and another function is for multi-channel distribution.
• Body: The content of the Body is the data sent. According to different tyoeID, the format is different.

The format block structure of the specific Message is as follows:

.
    +--------------+----------------+--------------------+--------------+
    | Basic Header | Message Header | Extended Timestamp |  Chunk Data  |
    +--------------+----------------+--------------------+--------------+
    |                                                    |
    |<------------------- Chunk Header ----------------->|
                                Chunk Format

Block Basic Header (1-3): This byte contains the block stream ID and the block type , which determines the number of encoded message headers. This field is a variable length field whose length depends on the block stream ID.
Message Header (0, 3, 7, 11): This field contains information about the message being sent . The field length is determined by the type of the block in the block header.
Extended Timestamp (0,4 bytes): The time this byte exists depends on the timestamp encoded in the block message header .
Block data (variable size): valid data of the current block, the maximum size of the block configured by the upper limit.

Basic Header (basic header information): contains stream ID and chunk type. Chunk stream ID is generally written as CSID, which is used to uniquely identify a specific stream channel. Chunk type determines the format of the following Message Header. The length of the basic header is 1, 2, and 3 bytes. The chunk type is fixed and occupies 2 bits. The length of the basic header depends on the size of the CSID. On the premise of enough storage for these two fields, use as few words as possible. Section thus reduces the amount of data increased due to the introduction of Header.

The RTMP protocol supports user-defined CSID between [3,65599], 0,1,2 are reserved for special message information by the protocol, 0 means that the Basic Header takes up 2 bytes in total, CSID is between [64,319], 1 means use 3 bytes, CSID between [64,65599], 2 means that the chunk is control information and some command information. 3 means Basic is 1 byte

• When the Basic Header is 1 byte
  

CSID is 6 digits, and 6 digits represent 64 numbers. In this case, it can only be between [0, 63], and the user can choose the range of [3, 63]

• When the Basic Header is 2 bytes

CSID occupies 14 bits. At this time, the protocol will set the other positions of the byte where the chunk type is located to be 0, and the remaining byte represents CSID-64. In this way, there are 8 binary bits to store CSID, and 8 bits can be expressed as [0,255 ] There are 256 numbers in total, so in this case [64,319], where 319=255+63

• When the Basic Header is 2 bytes

CSID occupies 22 bits. At this time, the protocol sets [2, 8] bytes to 1, and the remaining 16 bytes represent CSID-64, so there are 16 bits to store CSID, and 16 bits can represent a total of [0, 65535] 65536 numbers, so in this case the CSID is in [64, 65599], where 65599=65535+64, it should be noted that the Basic Header is stored in little endian, and the order of magnitude is higher as the number of bytes goes backwards, so When calculating the CSID by the value of each bit of these 3 bytes, it should be: <value of the third byte>x256+<value of the second byte>+64

• Message Header (the header information of the message)

Contains the description information of the actual information to be sent . The format and length of the Message Header depends on the chunl type of the Basic Header (fmt: 0, 1, 2, 3). There are 4 different formats, which are composed of the above-mentioned Basic The fmt field control in the Header . The first format can represent all the data of the other three representations , but since the other three formats are based on the representation of the difference quantization of the previous chunk, it can represent the same data more concisely, and should be used in actual use As few bytes as possible represent data with the same meaning . The following introduces the four formats in order of the number of bytes from most to least

fmt=0: (type=0)

When type=0 at that time, the Message is regarded as 11 bytes, and it can represent the other three data. But after the first chunk at the beginning of the chunk streams and the timestamp in the header information (the value is reduced compared with the previous chunk, usually when playing back, it will appear like this) when you must use this format

timestamp (time stamp): takes 3 bytes, so it can represent up to 16777215=0xFFFFFF=2^24-1, when its value exceeds this maximum value, these three bytes are all set to 1, so that the actual The timestamp will be transferred to the Extended Timestamp field, and the receiver will analyze the actual timestamp in the Extended timestamp when it judges that all 24 bits of the timestamp field are 1.

message length (the length of the message data): occupies 3 bytes, indicating the length of the data of the message actually sent, such as audio frames, video frames, etc., in bytes. Note that this is the length of the Message, that is, the total data length of the Message to which the chunk belongs, not the data length of the Data of the chunk itself.

message type id (message type id): Occupies 1 byte, indicating the type of data actually sent, such as 8 for audio data, 9 for video data.

msg stream id (message stream id): Occupies 4 bytes, indicating the ID of the stream where the chunk is located. Like the CSID of the Basic Header, it uses little-endian storage

fmt=1: (type=1)

When type=1, the Message Header occupies 7 bytes, omitting 4 bytes representing the msg stream id, indicating that this chunk is in the same stream as the chunk sent last time. When the sending end only has one stream connection with the peer end, it can try to adopt this format as much as possible.

timestamp delta: Occupies 3 bytes, note that this is different from when type=0, and stores the time difference from the previous chunk. Similar to the timestamp mentioned above, when its value exceeds the maximum value that can be represented by 3 bytes, all three bytes are set to 1, and the actual timestamp difference will be transferred to the Extended Timestamp
field, accept When the end judges that all 24 bits of the timestamp delta field are 1, it will go to the Extended timestamp to analyze the difference between the timing and the last timestamp.

fmt=2: (TYPE=2)

When type=2, the Message Header occupies 3 bytes. Compared with the type=1 format, 3 bytes representing the message length and 1 byte representing the message type are omitted, indicating The stream, message length, and message type of this chunk and the chunk sent last time are the same. The remaining three bytes represent timestamp delta, use the same type=1.

mt=3：（TYPE=3）：

0 bytes, indicating that the Message Header of this chunk is exactly the same as the previous one, so naturally there is no need to transmit it again.

When it follows the chunk of Type=0, it means that the timestamp of the previous chunk is the same. When are the timestamps the same? That is, a Message is split into multiple chunks, and this chunk belongs to the same Message as the previous chunk.
And when it follows the chunk of Type=1 or Type=2, it means that the difference with the timestamp of the previous chunk is the same. For example, Type=0, timestamp=100 of the first chunk, Type=2, timestamp delta=20 of the second chunk, indicating that the timestamp is 100+20=120, Type=3 of the third chunk, indicating timestamp delta =20, the timestamp is 120+20=140

• Extended Timestamp

As mentioned above, there will be a timestamp and a timestamp delta in the chunk, and they will not exist at the same time. This will only be used when one of the two is greater than the maximum value 0xFFFFFF=16777215 that can be represented by 3 bytes. Field to represent the real timestamp, otherwise this field is 0. The extended timestamp occupies 4 bytes, and the maximum value that can be represented is 0xFFFFFFFF=4294967295. When the extended timestamp is enabled, the timestamp field or timestamp delta must be set to 1, indicating that the timestamp field should be extended to extract the real timestamp or timestamp difference.

• Chunk Data (block data)
  the protocol-independent data that the user really wants to send, the length is between (0, chunkSize]