WebRTC is a streaming media engine widely used in WEB browsers. It realizes the transmission of audio and video data in a point-to-point manner to complete functions such as video conferencing. However, considering that WebRTC is mainly for point-to-point video conferencing services with a limited number of people, it is not suitable for other live broadcast applications or when accessing existing streaming media networks. You can consider using the WebRTC stream on the browser side , converted to other streaming media protocols such as RTMP/GB28181. After a period of research, this function has been successfully implemented. Here I will introduce the conversion principle.
The benefits of this article, free C++ audio and video learning materials package, technical video/code, including (audio and video development, interview questions, FFmpeg, webRTC, rtmp, hls, rtsp, ffplay, codec, push-pull stream, srs)↓↓↓ ↓↓↓See below↓↓Click at the bottom of the article to get it for free↓↓
WebRTC workflow
The WebRTC protocol mainly consists of three parts:
- User Interface: Provides a set of APIs that enable developers to communicate audio and video between browsers and mobile applications;
- Network Protocol Stack: Responsible for network data transmission and protocol processing;
- Audio/Video Engine: responsible for the processing and encoding and decoding of audio and video data.
Before audio and video communication is established, a series of interactions need to be performed between browsers through a signaling server to negotiate session parameters and communication methods. The following is the signaling interaction process of WebRTC:
- Establish a room (Room): Both browsers A and B need to enter the same room for audio and video communication. Before entering the room, they need to be authenticated and authorized by the signaling server.
- Exchange SessionDescriptionProtocol (SDP): After entering the room, both browsers A and B will generate SDP information to negotiate the parameters and format of audio and video communication. They will send their own SDP information to each other through the signaling server.
- Exchange ICECandidate: In order to penetrate NAT and firewall, WebRTC uses ICE protocol to obtain NAT type and external IP address. During the SDP negotiation process, browsers A and B exchange ICECandidate information for NAT penetration.
- Establish PeerConnection: After completing SDP and ICE negotiation, browsers A and B can establish PeerConnection and start audio and video transmission. PeerConnection will establish a point-to-point connection between browsers, and supports multiple transmission methods, including UDP, TCP, and SCTP.
When the signaling interaction is completed, the two sides of WebRTC establish a data channel and start to transmit data. WebRTC supports a variety of data transmission methods, including Real-time Transport Protocol (Real-time Transport Protocol, RTP), User Datagram Protocol (User Datagram Protocol, UDP), Transmission Control Protocol (Transmission Control Protocol, TCP) and Data Channel Protocol (Data Channel Protocol), etc.
Among them, RTP is the most commonly used audio and video transmission protocol of WebRTC, which is used to transmit audio and video data in real time. It is based on the UDP protocol and provides some additional functions, such as packet loss recovery, flow control and clock synchronization. WebRTC can also use the RTCP protocol for quality control and feedback, including indicators such as network latency, jitter, and packet loss.
The UDP protocol is suitable for transmitting real-time audio and video data because of its low latency and high throughput. However, there are also some problems in the UDP protocol, such as packet loss and disorder, which need to be solved through additional mechanisms. WebRTC uses some optimization techniques, such as forward error correction, retransmission and reconstruction, etc., to improve the quality and stability of audio and video transmission.
In addition to transmitting audio and video data, WebRTC also supports data channel protocols for transmitting arbitrary data between browsers. The data channel protocol is based on the SCTP protocol and supports functions such as reliable transmission and flow control. It can be used to transfer files, messages and game data, etc., providing a new way of communicating between browsers.
Protocol conversion scheme
Although WebRTC has many advantages, it does not have particularly obvious advantages for live broadcast and other fields, but one of the biggest advantages is that it can be used for camera acquisition, encoding and push on the browser. So you can take advantage of this to obtain the encoded camera data through the web page, and then further access it to other existing live broadcast protocols or workflows.
After understanding the entire workflow of WebRTC, you can set the protocol conversion scheme:
- First of all, it is necessary to construct a communication protocol for signaling communication with the browser, and monitor the data port, waiting for the browser to send WebRTC messages and data packets.
- When the browser page calls the WebRTC interface, first create RTCPeerConnection according to the normal process, and then create a local offer. After receiving the browser callback, extract the sdp information in the offer and upload it to the server
- After receiving the offer sent by the browser, the server parses the SDP packet, then generates local SDP data, fills in the SDP with relevant media information such as the local RTP port and RTCP port, and returns it to the browser.
- After the browser page receives the SDP, it calls the WebRTC interface, creates an answer, and sets the browser WebRTC module. If the SDP is correct, then the browser will start sending RTP messages to the server, including audio and video encoding data. , the program parses and extracts this, and repackages the relevant data with new protocols (RTMP, RTSP, GB28181).
Problems encountered and solutions
- It should be noted that when the browser communicates with the server, it will also send a STUN message, and the return packet of the STUN message needs to be processed properly, otherwise the problem of connection interruption will occur.
- RTCP packets must be processed and RTCP interactions of the normal process must be performed.
- The H.264 data encoded by WebRTC will only send a key frame once. If this is directly converted into a live stream of other protocols without processing, it is likely that the key frame cannot be obtained after the player is connected and cannot be decoded. This processing method is to send a key frame retransmission request to the browser when a key frame is needed. Of course it is also sent via RTCP.
The following is the format of the FIR (Full Intra Request) key frame retransmission message.
The benefits of this article, free C++ audio and video learning materials package, technical video/code, including (audio and video development, interview questions, FFmpeg, webRTC, rtmp, hls, rtsp, ffplay, codec, push-pull stream, srs)↓↓↓ ↓↓↓See below↓↓Click at the bottom of the article to get it for free↓↓