Keywords: smart city, security video, RTSP surveillance video webpage playback, 3D visualization, digital twin

1.1 Industry Pain Points

With the widespread development of intelligent applications across the country ( smart cities, smart communities, smart public security, smart fire protection, smart transportation, smart tourism, and smart education ), 3D visualization and digital twin platforms, as centralized display terminals for big data, have also gained popularity. Use in large areas.

Since the 3D visualization and digital twin platform, as a collection and display system of various big data, needs to access various business data, including various real-time security video signal sources, emergency communication video sources, industrial cameras, wireless Human-machine real-time return video, unmanned construction machinery cameras in smart mines, various real-time video signals on site during remote command, various robot visual images in industrial production, etc.

Based on the need to be compatible with the multi-terminal operating environment, the 3D visualization platform usually operates in the B/S mode. At this time, the traditional security video source [RTSP network stream] cannot be directly accessed on the browser side, but the RTSP network stream needs to be converted into HTML5 The browser-compatible stream protocol format can be played normally.

Faced with this situation, traditional manufacturers convert the RTSP network stream into the HTTP Live Streaming (HLS) protocol released by Apple through the streaming media server, and then realize multi-terminal browser playback. However, because the HLS protocol is a progressive segmentation The file download protocol is not a streaming media protocol in the true sense, so it inherently has the problem of long transmission delay. The lowest delay effect that can be achieved in the industry is usually about 3 seconds, but this delay index cannot meet specific application scenarios at all. real-time communication requirements.

There are also some manufacturers who convert RTSP network streams into HTTP-FLV format network streams for PC terminals, so as to achieve a network delay of 1 to 2 seconds, thereby further improving the real-time performance of the PC terminal. However, this still cannot meet the needs of application scenarios such as emergency communication, remote command and real-time interaction.

1.2 Solution Introduction

Based on the current pain points of industry applications, based on the streaming media technology research and development capabilities accumulated over the years, our company successfully developed a set of ultra-low latency and HTML5-compliant unified video access solutions at the beginning of 2020 over the past three years. Since the solution was launched on the market at the end of 2020, it has greatly improved the real-time communication experience, and controlled the end-to-end transmission delay to about 300ms in a private network environment, which has been greatly recognized by industry partners and end users. As an important technological innovation in the industry, this solution brings real value to end users.

Because this technology is fully compatible with the HTML5 standard, it can run normally on the PC side (including Windwos system, Linux system, localized operating system), Android device side, and iOS device side, and there is no need to install browser plug-ins from major monitoring manufacturers. , which greatly improves the user experience, and can be perfectly compatible with various business systems (3D visualization system, digital twin, GIS system).

1.3 Technical Implementation

1.3.1 Technical Architecture

The technical implementation architecture diagram of the system is as follows:

1.3.2 Functional module composition

The platform is mainly composed of a low-latency video transcoding workstation and a low-latency live broadcast publishing server .

Low-latency video transcoding workstation : used to achieve unified access to the security cameras of various front-end manufacturers, and realize unified protocol and encoding format conversion, and push it to the low-latency live broadcast publishing server in low-latency mode.

Low-latency live broadcast publishing server: used to realize low-latency forwarding of various network streams, publish in HTML5 for various terminal devices (PC, iOS devices, Android devices), and support one-to-many high-concurrency applications.

1.3.3 Supported terminal types

Existing solutions can support the following device terminals:

PC terminal

Android terminal

iOS terminal

OS type:

Windows/Linux/MacOS

Browser type:

Chrome/Firefox/Safari/Edge

Browser type:

Chrome/Firefox

WeChat, WeChat applets

Browser type:

Safari

WeChat, WeChat applets

1.3.4 Concurrency performance indicators

After actual testing, the concurrency performance indicators of our low-latency live server software system are as follows:

Server hardware configuration environment:

CPU：Intel E5-2650

Memory: 16GB

Hard Disk: 120GB SSD

Network card: Intel Gigabit LAN x 4 ports

Server OS:

CentOS x64 7.6

Live stream: 2Mb/s

Image resolution: 1280x720

Video encoding format: H.264

Concurrent performance index: 2000 concurrent live broadcast reception

Peak CPU usage: 42%

Average CPU usage: 35%

Average memory usage: 56%

1.3.5 Low Latency Specifications

The end-to-end delay of the system mainly occurs in the following links:

1. Video acquisition and encoding delay;

This part of the delay appears on the camera side, and the delay is in the range of 20~50ms;

2. Video access and transcoding delay;

This part of the delay occurs at the low-latency video transcoding workstation. When performing protocol conversion and video encoding format conversion, the delay is in the range of 10~30ms;

3. Delay in live release service;

This part of the delay occurs on the server side of the low-latency live broadcast publishing server. When the server receives the network stream pushed by the low-latency video transcoding workstation, it needs to cache 2~3 frames of data locally to resist network bandwidth jitter. To avoid the impact of the picture freeze.

According to different network stream formats, this part of the delay is in the range of 40~100ms;

4. Client decoding playback delay:

When the client-side HTML5 player is playing the network stream, it needs to wait for a complete frame of data to be received before it can decode and output, and it also needs to cache 1~2 frames of data based on the impact of fighting network jitter, so this part of the delay is in 20~80ms range;

To sum up, the delay time of the entire end-to-end system is usually in the range of 300~500ms, which is basically consistent with the browser plug-in mode of the monitoring manufacturer.