Because the project requires the front-end to play multiple live videos at the same time on the PC browser and mobile (android/ios browser) to play (mute), the resolution is not required, but the real-time performance is required, and you can choose the live broadcast in the live broadcast list at will Video, the corresponding live video window can be drawn on the right side (the number is not limited, the video I want to draw can no longer be used for live video, the hardware and network requirements are too high, the page will be stuck, so I use canvas to draw the video) , I haven’t done a live video before, first look for information and write various examples to summarize and finally find a relatively better way to achieve it, the record is as follows:
1. Introduction to Internet Streaming Media
1. HTTP stream is an http stream defined by each company, which is used in domestic video-on-demand websites. HTTP is used for on-demand, which is essentially file distribution and has poor real-time performance.
2. HLS is an HTTP-based streaming media transmission protocol implemented by Apple. The full name is HTTP Live Streaming. It has the advantages of high performance and perfect support for iOS. It can support live streaming and on-demand streaming of media. It is mainly used in the iOS system and is an iOS device. (Such as iPhone, iPad) provide audio and video live broadcast and on-demand programs.
Compared with the RTMP protocol, HLS does not need to install an APP on the mobile terminal, and is compatible with HTML5, so it has a huge advantage in the spread and experience of mobile live broadcast. However, the real-time performance of HLS is poor, and the industry's average live broadcast delay is 10s ~ 35s. It will cut the streaming media data into continuous short ts files on the server side, and access the ts files in order through the M3U8 index file. The client only needs to continuously play the files obtained from the server in order to realize the playback of audio and video.
Advantages of HLS:
Apple's full range of product support: Since HLS was proposed by Apple, Apple's full range of products including iPhone, iPad, and safari can natively support HLS without installing any plug-ins. Now Android has also added support for HLS. Through the firewall. Based on HTTP/80 transmission, effectively avoiding firewall interception
High performance. Through HTTP transmission, network distribution is supported, CDN support is good, and it comes with multi-rate adaptation. When Apple proposed HLS, it already considered the issue of code stream adaptation.
Disadvantages of HLS:
The real-time performance is poor, and the delay is high. The delay of HLS is basically above 10s+
File fragmentation. A double-edged sword of features, ts slices are small, which will cause a large number of small files, which poses certain challenges to storage and caching.
3. RTMP is a real-time messaging protocol, and Real Time Messaging Protocol is an open protocol developed by Adobe Systems for audio, video and data transmission between Flash players and servers. The protocol is based on TCP, which is a protocol family, including RTMP basic protocol and various variants such as RTMPT/RTMPS/RTMPE. RTMP is a network protocol designed for real-time data communication. It is mainly used for audio, video and data communication between the Flash/AIR platform and the streaming media/interactive server that supports the RTMP protocol.
RTMP is the application layer protocol in the Internet TCP/IP five-layer system structure. It is essentially a streaming protocol. The main advantage is high real-time performance. It can basically control the live broadcast delay within 3 seconds and has high stability, so it is widely used in low-level applications. Delayed live broadcast. It is mainly used for live broadcast applications and has certain requirements for real-time performance.
In live broadcast applications, RTMP and HLS are two relatively mature and widely used streaming media protocols, which can basically cover all clients.
The RTMP protocol generally transmits flv, f4v format streams, and the RTSP protocol generally transmits ts, mp4 format streams. HTTP does not have a specific stream.
RTMP's optimization of the bottom layer is better than other protocols. At the same time, it supports Adobe Flash well. Basically all encoders (cameras and the like) support RTMP output. The PC market is huge now, PCs are mainly Windows, and Windows browsers basically support Flash. It also has shortcomings. On the one hand, it is based on TCP transmission. Non-public ports may be blocked by firewalls. On the other hand, RTMP is a proprietary protocol of Adobe, which cannot be played by many devices, especially on the iOS side. Only three-party decoders can be played.
4. RTSP (Real-time Streaming Media Protocol) is an application layer protocol jointly proposed by Real network and Netscape for how to effectively transmit streaming media data on an IP network.
Browser does not support rtsp is not supported now, rtsp is converted to rtmp and hls to achieve cross-platform playback delay 3-5 seconds
Use the plug-in to convert to .flv or websoket ws: ws format
5. FLV (Flash Video) is another video format launched by Adobe. It is a storage container format for streaming media data transmitted on the network. Its format is relatively simple and lightweight, and does not require large media header information. The entire FLV is composed of The FLV Header, The FLV Body and other tags. Therefore, the loading speed is extremely fast. The suffix of the file encapsulated in FLV format is .flv.
2. The following are some of the solutions I find for realizing live video on the front-end, including plug-ins. I need to compare and select both a PC browser and a mobile terminal (Android/ios Google browser) and support multiple live video playback, and require real-time Sex, and need to use canvas to draw the video. If your needs are not high, you can choose the corresponding play program.
The following is a summary of the comparison and list after I wrote the case in combination with the agreement and the plug-in (the time difference between recording and writing the project may be long and there may be discrepancies, you can test according to the actual situation. I will slowly sort out the plug-in cases used in The link below, the plug-in case can also be searched by yourself):
1 、 HLS videos
pc: live video support
ios: Live video support, canvas drawing support, high latency
android: Live video is only supported by Google, canvas drawing is not supported, and the delay is high
2、rtsp jsmpeg+websocket
ws://192.168.1.34:8082/live2 jsmpeg plugin
ios: Live video support, canvas drawing support, low latency, cumulative play time, only 8 live videos can be opened at most
android: live video support, canvas drawing support, low latency, cumulative play time, only 8 live videos can be opened at most
rtsp real-time is best. rtsp jsmpeg PC supports 16 and ios 8 at most, and it is not synchronized. In Apple and low-performance Android, the lag is serious and accounts for traffic.
3、HDL（HTTP-FLV ） flv.js
ios: Live video is not supported. m3u8 can be played
android: .flv file live video is only supported by Google, canvas drawing support, low latency
Using flvjs on pc can be good
flv.js is restricted by Media Source Extensions API and is not compatible with IOS.
Bilibili also uses hls on the mobile h5 page
flv.js and video.js play .m3u8 files, ios and android are the same, ios can play video, but android cannot draw m3u8 and PC does not support it.
4. RTMP flash does not support
videojs + flash.js Google has been blocked, you need to open flash manually
Rtmp mobile terminal should not support flash, flash support is required, html does not support
5、ijkplayer
An open source library based on ffmpeg bilibili Android IikMediaPlayer ijkplayer is a lightweight Android/iOS video player based on FFmpeg flv.js pc
6、EasyPlayer.js
A web live/on-demand player that integrates rtmp, hls, flv, websocket, simple to use and powerful
EasyPlayer & EasyPlayerPro is a complete streaming media player project developed and maintained by the EasyDarwin team.
Currently supports Windows (supports multiple windows, includes ActiveX, npAPI Web plug-ins), Android platform, iOS platform,
Video supports H.264, H.265, MPEG4, MJPEG,
Audio supports G711A, G711U, G726, AAC,
Support RTSP over TCP/UDP, RTMP, HTTP, HLS and other protocols, support hard decoding,
PC web page test: up to 8, severely stuck
Cannot display rtmp. flash has been blocked, http://rtsp cannot be played
webSocket ws://192.168.1.34:8082/live2 can play
android play 4, freeze
9, python build flask server to play rtsp, rtmp streaming media
https://www.cnblogs.com/FHC1994/p/11724484.html
If there is a delay problem, multiple imgs can be played
3. Summary of own project
The final project went through two versions
1. The front end of the first version uses video.js to play live video of .m3u8. You can play multiple, PC and mobile terminals can play, but multiple videos are sending http request slice data, the more video sources, the page will be more stuck, the PC terminal is still smooth, the mobile terminal is stuck a little (the page is switched with swipe The video source, I have optimized, and the currently displayed video sends a request. For example, the page displays 5 videos. After the swiper arrow is switched, the video that cannot be seen will not be requested). The point is that we require real-time performance. The PC side has a better effect, but also has a 5s delay, and the mobile side has a higher delay. The project will be optimized after a period of use.
2. The front end of the second version uses jsmpeg+websocket to play rtsp live video. The real-time performance of rtsp is the best, but it can’t play too many video sources. I tested the pc with 16 at most, and it’s stuck when moving 8 of them. So if we want to use it, our solution is to use nodejs on the backend to convert the video of the rtsp protocol. Converted to ws: format
ffmpeg -i "你的rtsp" -q 0 -f mpegts -codec:v mpeg1video -s 800x600 http://127.0.0.1:8081/supersecret/live1node websocket.js supersecret 8081 8082'ws://192.168.0.59:9082/live1'The front end uses the jsmpeg plug-in for playback, which has good real-time performance and a delay of about 1s. As for the problem of multiple videos, the background stitched multiple video sources to one address, that is, I only need to visit one address to get all the video sources, and then select the video sources according to the coordinates, which guarantees that Real-time, and can optimize the request to play multiple video sources.
Four, related cases
1. video.js plays hls live video (.m3u8) (for the time being, only one live video is shown)
https://blog.csdn.net/qq_40015157/article/details/102611895
2. jsmpeg+websocket play rtsp to achieve low latency play live video (single, multiple)
https://blog.csdn.net/qq_40015157/article/details/113057166
3. Use VLC to convert MP4 to rtsp protocol video and play the detailed steps. The trsp address can be used as the test address in the project.
https://blog.csdn.net/qq_40015157/article/details/111238330
4. A small case of canvas drawing video
https://blog.csdn.net/qq_40015157/article/details/111208048
The follow-up code is being sorted out... If there are any discrepancies, please point out, don't spray, thank you.
Five, project screenshots
I only cut a small part of the display video source and operation area:
