Demand Analysis
The video streaming media monitoring industry has entered the Internet era, and browsers carry the vast majority of Internet access traffic. At present, the common practice of broadcasting RTSP streams on web pages is to convert RTSP into Internet live broadcast protocol RTMP or HLS; while RTMP protocol playback The Flash plug-in is required, and the FLV or HLS protocol derived from it has a large delay (more than 2s), which simply cannot meet the low-latency requirements of video streaming media transmission.
The Internet live broadcast RTMP protocol that was popular in the early years can only be supported by the flash browser plug-in player, while Flash does not support RTSP, and it is estimated that it will not support it in the future, and the FLASH plug-in has also been eliminated by browser manufacturers;
Then we can make browser plug-ins to play RTSP by ourselves. Through npapi, ppapi plug-ins, and ocx plug-ins for IE, the player executable program can be made into a plug-in for the browser to call; however, the compatibility is too poor and the development cost is too high. . For PC web compatibility, currently the best solution is flash or H5. For mobile web/WeChat compatibility, there is no doubt that H5 is the only choice;
Solution
How to solve several problems in the above requirements, the solution is as follows:
Play RTSP protocol directly through H5
On the PC side, the RTSP protocol with strong compatibility is output through streaming media, and the RTSP protocol is played directly through the WEBSOCKET interaction with H5, then the above problems are solved, that is, low latency is guaranteed and it can be played directly on the web page without plug-ins, which is simple and efficient; Simultaneously output: rtmp/hls/http-flv multiple streams, increase the compatibility and adaptation of the front end, and you can perfectly achieve the desired solution. In summary, you need to go through the following steps:
RTSP streaming;
audio and video transcoding (optional);
streaming media server RTSP forwarding + WEBSOCKET proxy; streaming media server
multi-protocol forwarding RTMP/HTTP-FLV/HLS/WS-FLV;
front-end H5 without plug-in streaming playback;
technical implementation
RTSP streaming
There are currently two solutions on the market that can pull RTSP streams from various manufacturers in a very compatible way:
Live555
Live555 has high real-time streaming performance, but poor compatibility, and may not be able to pull RTSP streams from some niche manufacturers or RTSP streams with poor standards;
FFmpeg
FFmpeg has high streaming stability, strong compatibility, and relatively low real-time performance. We can usually improve real-time performance by adjusting parameters.
Of course, both can request to obtain the camera streams of various manufacturers relatively well, but in terms of compatibility, stability, reliability, and operability, FFmpeg is even better.
There is no absolute, according to the needs, maybe you only need to connect two specific types of cameras, how to suit the needs of the scene;
2. Audio and video transcoding (H.265 to H.264, audio to AAC)
Due to the current WEB front-end H5 support, the support for H264 is better, for example: H264 supports hardware decoding, and the decoding efficiency is higher; while H265 only supports soft decoding, the decoding efficiency is relatively low; therefore, we need to convert various Video format: H.265, MJPEG, MPEG4 converted into H.264 and then forwarded to H5 for playback, various audio formats: G.711A/U, G.726, all transcoded into AAC format, the same reason, H5 AAC support is better; as for audio and video transcoding, the magic recognized by the industry is of course FFmpeg.
3. Streaming server RTSP forwarding + WEBSOCKET proxy
SkeyeSMS supports RTSP forwarding streaming media service. We refer to Live555's lightweight RTSPServer streaming media service design idea, fully absorb its ultra-low latency characteristics, and design a multi-threaded distribution strategy on this basis to improve the concurrency and distribution of RTSP streaming media distribution Efficiency, to make up for the lack of concurrency of Live555 single-threaded distribution. At the same time, the WEBSocket proxy algorithm strategy is added to share a distribution cache queue without affecting the original RTSPServer distribution strategy, and establish an efficient distribution channel that interacts with H5 to achieve the effect of playing RTSP with multiple pages and no plug-ins.
Regarding the time stamp optimization, the time stamps of the cameras of some surveillance manufacturers (Dahua, Xiongmai, etc.) are extremely uneven, which will cause the streams distributed by streaming media to often appear fast and slow. , The phenomenon of stuck buffer loading, so it is necessary to uniformize the time stamp once during streaming media distribution. In this regard, you can refer to the -re command scheme of ffmpeg to optimize the time stamp to ensure uniform playback;
4. Streaming media server multi-protocol forwarding RTMP/FLV/HLS
The RTMP server mentioned here has several output protocols:
* rtmp
* hls
* http-flv
* ws-flv
liveweb streaming service refers to nginx-rtmp-module streaming rtmp forwarding service, on this basis, it develops support for http-flv and ws-flv protocols, and removes its RTMP The streaming module avoids unreasonably increasing the delay of streaming forwarding, optimizes and improves the efficiency of streaming media forwarding, and realizes efficient, stable, and highly concurrent multi-streaming media protocol distribution;
5. The front end is compatible with streaming playback
The front-end of liveweb streaming media adopts the advanced front-end framework of VUE+elementUI, which is widely used in the industry, and can respond to requests from terminals on various platforms. It distributes PC web, mobile web, and WeChat to obtain RTSP from the streaming front end of the network camera and pass liveweb media The server forwards live streams such as rtsp, rtmp, hls, http-flv, ws-flv;
