GB 28181: Real-time video playback

process

A streaming media platform is mainly divided into several parts

Main purpose: docking surveillance cameras, live video, many-to-many video chat

The system architecture consists of three parts:

• The interface server (HTTP server) is mainly used to respond to client requests
• The signaling server (SIP server) is mainly used to interact with streaming media servers and video devices. Its main function is to manage devices such as cameras and control which port the camera forwards the video stream to the streaming media server.
• The streaming media server is mainly used to process the reception, forwarding and distribution of video streams

The interface server and signaling server can be integrated into one server.

It is best to deploy the streaming media server separately to avoid excessive pressure on the streaming media server, and ZLMediaKit can be used instead. What we want to develop is the interface server and signaling server.

The real-time stream on demand adopts the INVITE method in the SPI protocol to realize the session connection, and uses the RTP/RTCP protocol to realize the media transmission.

The whole playback process is as follows:

Its communication process is as follows:

How the video stream is pushed

The acquisition of SIP video stream refers to the process that the decoder (ZLMediaKit) obtains the video stream from the GB28181 server through the SIP protocol

• The GB28181 server (that is, the SIP server) will send some signaling like the camera, mainly to request to play real-time video.
  • Open the SIP server, the server will listen to a certain port, and you can receive SDP information from this port
  • After the camera sends the Register message, the server responds with the registration message code
  • After the SIP server responds to the registration command, it sends an Invite request to request catalog information, which is the basic information of the device.
  • The camera end receives the message and sends its own catalog message, and will serve specific specific catalog messages.
  • After the SIP service fetches the information of the camera, it can send the requested video information to the camera. Let's see how to write SDP information

static string createSDP(MediaContext& mediaContext)
{
    
    
	char str[500] = {
    
     0 };
	pj_ansi_snprintf(str, 500,
	"v=0\n"
	"o=%s 0 0 IN IP4 %s\n"
	"s=Play\n"
	"c=IN IP4 %s\n"
	"t=0 0\n"
	"m=video %d RTP/AVP 96 98 97\n"
	"a=recvonly\n"
	"a=rtpmap:96 PS/90000\n"
	"a=rtpmap:98 H264/90000\n"
	"a=rtpmap:97 MPEG4/90000\n"
	"y=0100000001\n",
	mediaContext.GetDeviceId().c_str(),
	mediaContext.GetRecvAddress().c_str(),
	mediaContext.GetRecvAddress().c_str(),
	mediaContext.GetRecvPort()
			);
	return str;
}

• After the signaling interaction is successful, the camera (media stream sender) will push the video data to the specified port in rtp mode (the port is specified in the above invite message) and stream it to the media server
• The media server (GB28181 server) receives the video stream at the specified port
• The media server (GB28181 server ) forwards the video stream to the streaming receiver (such as ZLMediaKit)

The specific process is as follows:

video stream format

GB28181 requires the transmitted video stream format to be PS stream, or H264 stream, or MP4 format.

You can use wireshark to capture packets, and the datagram type is RTP PS stream

The national standard streaming media server is actually responsible for converting PS streams pushed by GB28181 devices or platforms into ES streams, and then providing RTSP, RTMP, FLV, HLS and other formats for distribution.

The difference between PS stream and ES stream

An IP datagram consists of a header and data. The first part of the header
has a fixed length of 20 bytes, which is a must for all IP datagrams. The header includes: total length, logo, MF, DF, slice offset.
What the digital signal actually transmits is a data stream, and the general data stream includes the following three types:

• ES stream (Elementary Stream): also called elementary stream, which contains continuous stream of video, audio or data.
• PES stream (Packet Elementary Stream): also called packaged basic code stream, which divides the basic code stream ES stream into data packets of different lengths according to needs
  , and adds the packet header to form a packaged elementary code stream PES stream.
• TS stream: Also called transport stream, it is composed of packets with a fixed length of 188 bytes and contains one or more programs with independent time bases. A program can also contain multiple video, audio, and text information ES streams; each ES stream will have a different PID mark, in order to analyze these ES streams, TS has some fixed PIDs to send program and ES stream information tables at intervals: PAT and PMT tables. Suitable for environments with many bit errors

ES is the data stream directly from the encoder, which can be an encoded video data stream, audio data stream, or other encoded data streams. After the ES stream passes through the PES packager, it is converted into a PES package, and then distributed through RTSP, RTMP, FLV, and HLS formats to realize playback on multiple terminals such as WEB, mobile phones, PCs, and WeChat.

Propagation mode

The GBT28181 protocol stipulates that the code stream uses RTP packet load, and it is recommended to be PS stream or ES stream. For the transmission of media stream, the active tcp and passive tcp methods are added on the basis of the original UDP transmission.

(1) UDP passive

• This is a common transmission method.
  • The GB28181 streaming media server listens to a single UDP port, and then sends a SIP signaling (INVITE), which contains the port for receiving media in the SDP
  • After the device receives the signaling, it analyzes the port, and then the device actively sends the video stream to the port monitored by the streaming media server through UDP.

(2) TCP passive

• There are two types, one is active and the other is passive
• Active: The device informs the server of its own media stream tcp port, and the server actively connects to the port on the device to obtain data. . This scenario is rarely used and can be ignored
• For passive: the streaming media server listens to a single TCP port, and then tells the device port through SIP signaling (INVITE), and the device actively sends video streams to the current streaming media server, basically equivalent to UDP streams

live video

process

Premise: Register successfully >>>>>>Heartbeat success>>>>>>Device catalog query>>>>>Real-time video viewing

server steps

Regardless of whether it is viewed by TCP or UDP, the steps are:
(1) Open the video port
(2) Send a real-time video request
(3) Wait for the device to reply 200OK
(4) Send ACK
(5) Play the code stream
(6) stop video request
(7) close video port
(8) normal wait

Packet analysis

Test equipment IP: 192.168.0.107
Server IP: 192.168.0.60

Real-time video creation_UDP

Step 0: [Server] Open the video port

Step 1: [Server >> Client] Request to play video

INVITE sip:34020000001310000002@4401020049 SIP/2.0
Call-ID: helloVideo
CSeq: 1 INVITE
From: <sip:44010200492000000001@4401020049>;tag=bccedfd0111
To: <sip:34020000001110000001@4401020049>
Max-Forwards: 70
Contact: <sip:34020000001310000002@4401020049>
Via: SIP/2.0/UDP 192.168.0.107:5060;branch=z9hG4bKee5c5d98-bff9-4f3000002
Content-Type: application/sdp
Content-Length: 225

v=0
o=34020000001310000002 0 0 IN IP4 192.168.0.60
s=Play
c=IN IP4 192.168.0.60
t=0 0
m=video 6000 RTP/AVP 96 98 97
a=recvonly
a=rtpmap:96 PS/90000
a=rtpmap:98 H264/90000
a=rtpmap:97 MPEG4/90000
y=0100000001
f=

Step 2: [Client >> Server]

Reply 101 first

SIP/2.0 100 Trying
Via: SIP/2.0/UDP 192.168.0.107:5060;branch=z9hG4bKee5c5d98-bff9-4f3000002
From: <sip:44010200492000000001@4401020049>;tag=bccedfd0111
To: <sip:34020000001110000001@4401020049>;tag=5f906952
Call-ID: helloVideo
CSeq: 1 INVITE
Server: Happytime Agent Ver 1.0
Content-Length: 0

Reply 200

SIP/2.0 200 OK
Via: SIP/2.0/UDP 192.168.0.107:5060;branch=z9hG4bKee5c5d98-bff9-4f3000002
From: <sip:44010200492000000001@4401020049>;tag=bccedfd0111
To: <sip:34020000001110000001@4401020049>;tag=5f906952
Contact: <sip:34020000001110000001@4401020049>
Call-ID: helloVideo
CSeq: 1 INVITE
Max-Forwards: 70
Allow: ACK,BYE,CANCEL,INVITE,NOTIFY,REFER,UPDATE,INFO
Supported: timer
Session-Expires: 200;refresher=uac
Server: Happytime Agent Ver 1.0
Content-Type: application/sdp
Content-Length: 153

v=0
o=34020000001110000001 0 0 IN IP4 192.168.0.107
s=Play
c=IN IP4 192.168.0.107
t=0 0
m=video 19002 RTP/AVP 96
a=rtpmap:96 PS/90000
a=sendonly

Step 3: [Server>>Client] reply ACK

ACK sip:34020000001310000002@4401020049 SIP/2.0
Call-ID: helloVideo
CSeq: 1 ACK
From: <sip:44010200492000000001@4401020049>;tag=bccedfd0111
To: <sip:34020000001110000001@4401020049>
Max-Forwards: 70
Via: SIP/2.0/UDP 192.168.0.107:5060;branch=z9hG4bKee5c5d98-00003
Content-Length: 0

Step 4: Play code stream

Step 5: [Server >> Client] Stop the video request

BYE sip:34020000001310000002@4401020049 SIP/2.0
From: <sip:44010200492000000001@4401020049>;tag=bccedfd0111
To: sip:34020000001110000001@4401020049;tag=5f906952
CSeq: 2 BYE
Call-ID: helloVideo
Via: SIP/2.0/UDP 192.168.0.107:5060;branch=z9hG4bKee5c5d98-00004
Max-Forwards: 70
Content-Length: 0

Step 6: [Client] Response 200

SIP/2.0 200 OK
Via: SIP/2.0/UDP 192.168.0.107:5060;branch=z9hG4bKee5c5d98-00004
From: <sip:44010200492000000001@4401020049>;tag=bccedfd0111
To: sip:34020000001110000001@4401020049;tag=5f906952
Call-ID: helloVideo
CSeq: 2 BYE
Server: Happytime Agent Ver 1.0
Content-Length: 0

Step 7: [Server] Close the video port

Real-time video establishment_TCP

---

basic requirements

Signaling process

There are two signaling processes for real-time video streaming on demand:

• Client initiates
• Third Party Call Control

The networked system can choose one or a combination of the two implementation methods, and the third-party controller of the third-party call control should use back-to-back user agent implementation

The client initiates the process of real-time audio and video on demand

process

message demo

Packet analysis

After [upper-level platform] has subscribed to all channels, you can send Invite request flow to [lower-level platform]

INVITE sip:33072752001320080039@10.45.255.10:5060 SIP/2.0
Via: SIP/2.0/UDP 10.45.255.11:5060;rport;branch=z9hG4bK-469C18DE-28C36C5-T07h4NhO
From: <sip:33072700232000002456@3307270023>;tag=573748381
To: <sip:33072752001320080039@10.45.255.10:5060>
Call-ID: 743791906
CSeq: 20 INVITE
Contact: <sip:33072700232000002456@10.45.255.11:5060>
Content-Type: application/sdp
Max-Forwards: 70
User-Agent: videosvr 1.0
Subject: 33072752001320080039:1,33072700232000002456:1
Content-Length:   220

v=0
o=33072752001320080039 0 0 IN IP4 10.45.255.11
s=Play   // play表示播放
c=IN IP4 10.45.255.11
t=0 0
m=video 31000 RTP/AVP 96 98 97
a=recvonly
a=rtpmap:96 PS/90000
a=rtpmap:98 H264/90000
a=rtpmap:97 MPEG4/90000
y=0727000001

[The lower-level platform is (signaling server)] The response is as follows:

SIP/2.0 181 Call is being forwarded  // 181表示我已经将请求转发给下级了
CSeq: 20 INVITE
Call-ID: 743791906
From: <sip:33072700232000002456@3307270023>;tag=573748381
To: <sip:33072752001320080039@10.45.255.10:5060>
Via: SIP/2.0/UDP 10.45.255.11:5060;rport=5060;branch=z9hG4bK-469C18DE-28C36C5-T07h4NhO;received=10.45.255.11
Content-Length: 0

How to order

The basic process of on-demand

• HTTP side—>GB28181 signaling server, initiate an on-demand request
• GB28181 signaling server -> device, initiates an Invite (carrying SDP message body) request
• Device—>GB28181 signaling server, reply 200OK (carry SDP message body)
• GB28181 signaling server -> device, send ACK, successfully establish a session
• Device —> Streaming server, send live stream
• Streaming server —> GB28181 server, stream change event
• GB28181 server—>device, inform the stream playback address
• Streaming server --> GB28181 server, no one watching events
• GB28181 server—>device, send Bye message
• Device—>GB28181 server, reply 200OK

Possible errors on demand

(1) 400 error

The process is as follows:

• HTTP side—>GB28181 signaling server, initiate an on-demand request
• GB28181 signaling server -> device, initiates an Invite (carrying SDP message body) request
• Device—>GB28181 signaling server, reply 400OK (carry SDP message body)

what reason:

• The device thinks that the signaling server sent a wrong message to it, which may be incomplete or wrong, so it directly returns 400

How to deal with it:

• grab bag
• Contact the device/platform customer service directly for solutions

(2) 500 error

what reason:

• 500<= code <= 600 error code, generally there is a problem inside the device

How to deal with it:

How to deal with it:

• grab bag
• Contact the device/platform customer service directly for solutions

timeout on demand

There are roughly two reasons for on-demand timeout: on-demand timeout and stream receiving timeout.

(1) On-demand timeout

Where did it happen?

• On-demand timeout is generally due to signaling timeout, which may appear in the step of "device -> GB28181 signaling server, reply 200OK (carrying SDP message body)", that is, we sent an on-demand message, but the device did not reply.

possible reason:

• Device internal error, failed to reply message
• Network reason message did not reach the device

(2) Stream receiving timeout

Where did it happen?

• Device —> Streaming server, send live stream
• Streaming server —> GB28181 server, stream change event

possible reason:

• The message is sent but to the wrong IP and port

• Device internal error failed to send stream

• The device has sent the flow, but the flow cannot be identified. There may be cases where the flow is not standardized and the network is poor.

• The device sent the stream, and the streaming server also received it, but it could not notify wvp through the hook, probably because the network was not connected

• The device sent the stream, but the SSRC check was enabled, and the stream of the device was not standardized enough to use the wrong SSRC, causing the streaming media server to choose to discard it.

• Several common signaling processes in the GB28181 protocol

• GB28181 protocol – real-time video and audio on demand (preview)

• Video Surveillance Security Platform (Enterprise Level) - National Standard 28181 Platform

• gb28181 protocol

• document interpretation

• Live streaming on demand process

• sending process