As a front-end boy, have you known WebRTC?

WebRTC (Web Real-Time Communications) is an open source cross-platform real-time communication technology, we can use it to establish a peer-to-peer connection (peer-to-peer) in the browser without an intermediary, to achieve audio Video transmission, screen sharing, P2P file sharing and other functions.

WebRTC was open sourced on June 1, 2011 and was included in the World Wide Web Consortium's W3C standard with the support of Google, Mozilla, and Opera.

1. Compatibility

caniuse WebRTC

Modern browsers and mobile devices already fully support WebRTC.

2. WebRTC Architecture

The WebRTC architecture is divided into four layers

2.1 WebRTC C++ API layer

WebRTC C++ API (PeerConnection) realizes P2P connection, audio and video collection, audio and video transmission, non-audio and video data transmission, etc.

2.2 Session management layer

Session management / Abstract signaling (Session) session management layer, used to manage audio and video, non-audio and video data transmission, and process related logic.

2.3 Engine & transport protocol layer

Audio Engine (Voice Engine)

- iSAC/iLBC (Audio Codec)
- NetEQ for voice (handles network jitter and voice packet loss)
- Echo Canceler / Noise Reduction (Echo Cancellation / Noise Reduction)

Video Engine

- VP8 (video codec)
- Video jitter buffer (video jitter buffer, dealing with video jitter or video data loss caused by network reasons)
- Image enhancements (image enhancement module, improve video picture quality)

Transport protocol (Transport)

- SRTP (secure real-time transport protocol, which adds a security mechanism on the basis of RTP real-time transport protocol, similar to the relationship between HTTP and HTTPS)
- Multiplexing (multiplexing, multiple streaming data transmissions share one channel to improve transmission efficiency)
- P2P STUN+TURN+ICE (NAT penetration and relay service)

2.4 Hardware Adaptation Layer

Hardware adaptation layer, including: audio capture and rendering, video capture, network I/O. Browser manufacturers can implement it by themselves. This design increases the flexibility and cross-platform of WebRTC.

3. The core API of WebRTC

MediaStream API

- navigator.mediaDevices.getUserMedia (call local camera and microphone )
- navigator.mediaDevices.getDisplayMedia ( calls screen capture)

RTCPeerConnection API (establish peer connection)
RTCDataChannel API (transfer data between peer connections)

When calling mediaDevices to obtain media streams, the protocol of the current webpage must be localhost or https, otherwise navigator.mediaDevices is undefined.

4. WebRTC establishes a peer-to-peer connection

The premise of establishing a peer-to-peer connection: both parties must know the other party's IP address, supported media types, audio and video encoding and other information.

Local connection: If the two devices are in the local area network, there is no need to do any processing, just use the signaling server to obtain the other party's sdp, and then they can perform peer-to-peer connection through the LAN IP.

Public network connection: Except local connection. To realize P2P communication, both communication parties must know each other's public network IP.

4.1 How to obtain the public network IP?

In practical applications, due to reasons such as NAT and firewalls, we cannot directly obtain the public network IP of the machine. If we want to obtain the corresponding public network IP of the machine, NAT penetration is required.

NAT (Network Address Translation, Network Address Translation ) can convert the IP address in the private network to the public network IP address, so as to realize the communication between the internal network and the external network. NAT technology was originally designed to solve the problem of insufficient IPV4 addresses.

Generally, the WAN port of the router has a public network IP, and all devices connected to the router's LAN port will be assigned an IP address of a private network segment (for example, 192.168.1.2). The IP of the internal network device is mapped to the public network IP of the router and the only port. In this way, there is no need to assign a different public network IP to each internal network device, but it can still be discovered by external network devices.

Methods of NAT traversal:

STUN
TURN
ICE

4.1.1 STUN

STUN (Session Traversal Utilities for NAT, NAT session traversal application). Used to solve network address translation (NAT) problems. When a WebRTC connection is established between two devices, if there is NAT between them, you need to obtain your own public IP address and port number through the STUN server, so that the other party can directly connect to itself. The STUN server is only used to obtain the public network IP address and port number, and will not be used as a data transfer site.

The STUN server needs to implement itself. Free STUN service, eg: stun:stun.l.google.com:19302.

STUN cannot pass through symmetric NAT and firewall.

4.1.2 TURN

TURN (Traversal Using Relay NAT, relay server). It is used to solve the situation where direct connection through P2P is not possible. When a WebRTC connection cannot be established directly between two devices, it is necessary to use the TURN server as a relay site to transfer data from one device to another. The TURN server can transfer data between different networks for real-time communication between two devices.

Open source TURN service: coturn

4.1.3 ICE

ICE (Interactive Connectivity Establishment, interactive connection establishment) ICE is a negotiation framework. ICE will automatically select P2P or relay transmission according to factors such as network conditions and device capabilities to ensure the stability and quality of WebRTC connections.

ICE connection establishment process: ICE will try end-to-end direct connection first, and if the connection fails, it will call the STUN service to perform NAT penetration to obtain the public network IP, and if the connection fails, it will use the TURN server for relay transmission.

[Learning Address]: Advanced Development of FFmpeg/WebRTC/RTMP/NDK/Android Audio and Video Streaming Media
[Article Benefits]: Receive more audio and video learning materials packages, Dachang interview questions, technical videos and learning roadmaps for free, including ( C/C++, Linux, FFmpeg webRTC rtmp hls rtsp ffplay srs, etc.) If you need it, you can click 1079654574 to join the group to receive it~

4.2 How to get the media type, audio and video encoding?

Call RTCPeerConnection.createOffer/ RTCPeerConnection.createAnswerobtain the SDP of the current device.

SDP (Session Description Protocol) is a protocol that describes media negotiation information, including: media type, codec format, encryption algorithm, transmission protocol, etc. When two devices establish a WebRTC connection, SDP needs to be negotiated and exchanged so that they can understand each other's media parameters and protocol information.

Technically speaking, SDP is not a real protocol, but a data format used to describe the parameters and configuration information of peer-to-peer connections between devices.

Structure: An SDP consists of one or more lines of UTF-8 text, each line beginning with a character of the type, followed by an equals sign ("="), followed by structured text containing a value or description, formatted depending on the type. Lines of text beginning with a given letter are often called "letter lines". For example, a row that provides a media description is of type "m" and is called an "m row".

The following figure is an example of SDP:

5. Signaling server

The above mentioned how to obtain public network IP, media type, audio and video encoding and other information. So how do we inform the two parties of the peer-to-peer connection with this information?

We can achieve this through signaling. The process of sending device information between devices to determine the communication protocol, media codec, data transmission method, and any required routing information is called signaling (Signaling) .

The server that transmits signaling is called a signaling server. The signaling server does not perform special processing on the received information, but is only responsible for receiving and sending data.

5.1 The role of the signaling server

Exchange SDP
Send control messages: used to set up, open, close communications or handle exceptions.
Media capability negotiation: which codecs and media data formats can be supported by both communicating parties

After the SDP exchange, the WebRTC peer-to-peer communication really starts.

5.2 Build a signaling server

The signaling server does not belong to the WebRTC specification and needs to be built by ourselves. This can be done via WebSocket or polling HTTP requests.

6. webRTC communication process

Call RTCPeerConnection to create a peer connection
Call getUserMedia to open camera & microphone
Call RTCPeerConnection.addTrack to add the audio and video track to the peer-to-peer connection media stream track (it will be transmitted to the peer after the peer-to-peer connection is successful)
Call RTCPeerConnection.createOffer to create an SDP offer and call the setLocalDescription method to add sdp to the local description
Client A calls the ICE service to obtain the public network IP and generates an ICE candidate (ICE candidate)
Client A sends an offer to client B through the signaling server
Client B calls the setRemoteDescription method, sets the received SDP offer as a remote description, and displays the obtained audio and video streams
Client B calls getUserMedia to open the camera & microphone
Client B calls RTCPeerConnection.createAnswer to create an SDP answer, and calls the setLocalDescription method to add sdp to the local description
Client B sends SDP answer to client A through the signaling server
After client A receives it, it calls the setRemoteDescription method, sets the received SDP answer as a remote description, and displays the obtained audio and video stream
So far the connection is established

7. Application of WebRTC

Face recognition in credit business
call center system
Live broadcast, online education
WebRTC combined with TensorFlow to realize liveness detection
China Mobile Online Cancellation of Mobile Number Service
The bank's self-service card opening machine connects remote salespersons to online video card processing
Video double recording in the financial industry
IoT Internet of Things: Whole-house intelligence, smart speakers, home security cameras, phone watches, and more