Introduction to WebRTC technology, how to use webrtc and display of project code running effects

Ten thousand live conference-webrtc-srs-rtmp

WebRTC , namely Web Real-Time Communication , web real-time communication technology. Simply put, it is to introduce real-time communication in the web browser, including audio and video calls.

Introduction of WebRTC real-time communication technology

how to use

Media introduction

Signaling

Introduction to STUN and TURN

Peer connection and offer/answer negotiation

Data channel

NAT and firewall penetration

Simple application

other

Introduction of WebRTC real-time communication technology

WebRTC implements web-based voice conversations or video calls, with the goal of real-time communication on the web without plug-ins.

WebRTC provides the core technology of video conferencing, including audio and video capture, codec, network transmission, display and other functions, and also supports cross-platform, including linux, windows, mac, android, etc.

1. WebRTC triangle

2. WebRTC trapezoid

3. WebRTC multi-party conversation

WebRTC supports multi-party sessions or conference sessions involving multiple browsers. There are two modes for establishing such sessions:

4. New features of WebRTC

How to use WebRTC

WebRTC is easy to use, and media sessions can be established in a few steps. Some messages flow between the browser and the server, and some flow directly between the two browsers (becoming peers).

1. Establish a WebRTC session

The following steps are required to establish a WebRTC connection:

Get local media ( getUserMedia(), MediaStream API )

Establish a peer-to-peer connection ( RTCPeerConnection API ) between the browser and the peer (other browser or terminal )

Associate media and data channels to the connection

Exchange session description ( RTCSessionDescription )

Browser M requests a web page from the web server

The web server returns a web page with WebRTC js to M

Browser L requests a web page from the web server

The web server returns a web page with WebRTC js to L

M decides to communicate with L, and sends M's session description object (offer, proposal) to the Web server through M's own js

The web server sends the session description object of M to js on L

Js on L sends L’s session description object (answer, response) to the Web server

The web server forwards the response to js on M

M and L begin to interact and determine the best way to access each other

After completion, M and L begin to negotiate the communication key

M and L begin to exchange voice, video or data

The specific call flow of the WebRTC triangle session:

说明：
    SDP对象的传输可能是一个来回反复的过程，并且该过程采用的协议并未标准化

The specific calling process of the WebRTC ladder conversation mode:

说明：
    此场景中，浏览器M和L直接交换媒体，只是它们运行的Web服务器不用而已。每个浏览器的会话描述对象都会映射至Jingle[XEP-0166]session-initiate消息和session-accept方法。

Media introduction

First look at the local media in WebRTC:

1. Media in WebRTC

Track ( MediaStreamTrack , represents a single type of media that can be returned by a device or recorded content, and is uniquely associated with a "source". WebRTC cannot directly access or control the "source". All control of the "source" is implemented through the track; a "source" May correspond to multiple track objects)

Stream ( MediaStream , collection of track objects)

The schematic of the track and flow is as follows:

2. Capture local media

The following code shows the simple acquisition and display of local media:

// 注意getUserMedia()在各浏览器中的区别  
// Opera --> getUserMedia  
// Chrome --> webkitGetUserMedia  
// Firefox --> mozGetUserMedia  
navigator.getUserMedia = navigator.getUserMedia || navigator.webkitGetUserMedia || navigator.mozGetUserMedia;  
  
// 只获取video:  
var constraints = {audio: false, video: true};  
var video = document.querySelector("video");  
  
function successCallback(stream) {  
    // Note: make the returned stream available to console for inspection  
    window.stream = stream;  
      
if (window.URL) {  
        // Chrome浏览器
        video.srcObject = stream;  
    } else {  
        // Firefox和Opera: 可以直接把视频源设置为stream  
        video.src = stream;  
    }  
    // 播放  
    video.play();  
}  
  
function errorCallback(error){  
    console.log("navigator.getUserMedia error: ", error);  
}  
  
navigator.getUserMedia(constraints, successCallback, errorCallback);

The operation effect is as follows:

View the complete code: please add group 973961276

Signaling

In WebRTC, signaling plays a pivotal role. But the implementation is not standardized, such as http, websocket, xmpp, etc.

1. The role of signaling

Negotiate media features and settings

Identify and verify the identity of session participants (exchange information in SDP objects: metadata such as media type, codec, bandwidth, etc.)

Control media sessions, indicate progress, change sessions, terminate sessions

Double occupancy decomposition

Simply put, signaling is the process of coordinating communication. Once the signaling service is established and a connection is established between the two clients, they can theoretically communicate point-to-point.

2. Transmission of signaling

WebRTC requires the establishment of a two-way signaling channel between two peers. There are usually three ways to transmit WebRTC signaling: http, websocket, data channel

The http method is as follows:

Websocket proxy signaling transmission:

3. Server in WebRTC

WebRTC provides P2P communication on the browser side, but it does not mean that WebRTC does not require a server. Apart from application servers, at least the following two servers are required:

A server (signaling service) that exchanges various metadata (signaling) before communication is established between browsers

Servers that traverse NAT and firewalls (stun, turn, rsip, etc.)

说明：
    元数据是通过信令服务器中转发给另一个客户端，但是对于流媒体数据，一旦会话建立，首先尝试使用点对点连接。简单一点说就是：每个客户端都有一个唯一的地址，他能用来和其他客户端进行通讯和数据交换。
    
    STUN服务器：用来取外网地址的。（见下节）
    
    TURN服务器：在P2P失败时进行转发的。（见下节）
    
    ICE：*Interactive Connectivity Establishment*，即交互式连通建立方式。并非一种新的协议，它通过综合利用现有NAT穿透协议，以一种更有效的方式来组织会话建立过程，使之在不增加任何延迟同时比STUN等单一协议更具有健壮性、灵活性。

4. Signaling interaction and establishment of RTCPeerConnection

WebRTC uses RTCPeerConnection to establish a connection to transmit stream data. After establishing an RTCPeerConnection instance, to establish a point-to-point channel, two things need to be done:

Determine the characteristics of the media stream on the machine, such as resolution, encoding and decoding capabilities (SDP descriptor)

The network addresses of the hosts at both ends of the connection (ICE Candidate)

Exchange SDP descriptors through offer and answer:

A and B each create a PC instance

A establishes an offer signal containing A’s SDP descriptor through the createOffer() method provided by the PC

A uses the setLocalDescription() method provided by the PC to give A’s SDP descriptor to A’s PC instance

A sends the offer signaling to B through the server

B extracts the SDP descriptor contained in A’s offer signaling, and delivers it to B’s PC instance through the setRemoteDescription() method provided by the PC

B uses the createAnswer() method provided by the PC to create an answer signaling containing the SDP descriptor of B

B uses the setLocalDescription() method provided by the PC to give B’s SDP descriptor to B’s PC instance

B sends the answer signaling to A through the server

After A receives B’s answer signaling, he extracts B’s SDP descriptor and calls the setRemoteDescripttion() method to A’s own PC instance

Establish a NAT/firewall traversal connection through the ICE framework:

WebRTC uses the ICE framework to obtain the address that can be directly accessed by the outside world. RTCPeerConnection can pass in the address of the ICE server when it is created, such as:

var iceServer = {
    "iceServers": [{
        "url": "stun:stun.l.google.com:19302"
    }]
};
var pc = new RTCPeerConnection(iceServer);

A and B each create a PC instance configured with an ICE server, and add an onecandidate event callback to it

When the network candidate is available, the oneccandidate function will be called

Inside the callback function, A or B encapsulates the network candidate message in the ICE Candidate signaling, transfers it through the server, and delivers it to the other party

When A or B receives the ICE Candidate signaling sent by the other party through the server relay, it parses it and obtains the network candidate, and adds it to the PC instance through the addIceCandidate() method of the PC instance

In this way, the connection is created, and the media stream data can be transmitted by adding the stream to the RTCPeerConnection through addStream().

Introduction to STUN and TURN

The browser is located behind the network address translation device (NAT) is an extremely common design. Give a chestnut:

Look at the picture again, and understand the "public address" and "private address":

NAT is mainly responsible for maintaining the mapping table between internal ip address and port number and external ip address and port number.

1. STUN server

STUN, Session Traversal Utilities for NAT , is called the NAT session traversal utility server. Simply put, it is to obtain the outermost NAT (public ip address) information of the internal network device.

2. TURN server

TURN, Traversal Using Relay around NAT , is called a relay NAT traversal server.

说明：
    媒体中继地址是一个公共地址，用于转发接收到的包，或者将收到的数据包转发给浏览器。如果两个对等端因为NAT类型等原因不能直接建立P2P连接的话，那么可以使用中继地址。
    
    ps：相比较直接使用web服务器提供媒体中继理想点。

Peer connection and offer/answer negotiation

The previous section briefly introduced the peer-to-peer connection and the offer/answer interaction process, which will be explained in this section.

In fact, WebRTC defines two main functions, namely: media capture ( getUserMedia() , introduced earlier) and media transmission. The concept of peer-to-peer connection and offer/response negotiation is the core of media transmission.

1. Peer-to-peer connection

The RTCPeerConnection interface is the main API of WebRTC, used to establish media connections and data connection paths on the P2P side. The constructor of the RTCPeerConnection object has a series of attributes, the most important of which is the iceServers attribute, which represents a list of server addresses. Used to help establish a session through NAT and firewall.

var pc = new RTCPeerConnection({
    iceServers: [{
        url: 'stun:stun.l.google.com:19302'
    },{
        url: 'turn:[email protected]',
        credential: 'test'
    }]
})

getUserMedia({
    audio: true,
    video: true
}, successCB, failureCB)

function successCB(stream) {
    // 告知浏览器，我要发送MediaStream
    pc.addStream(stream)        // removeStream()
}

2. Proposal/response negotiation

To establish a connection between the two, a session must be established between the two. Offer/answer is a "one-time pass" negotiation mechanism. In practice, this process may be repeated many times.

WebRTC uses RTCSessionDescription objects to represent proposals and responses. Each browser will generate this object.

3. JavaScript proposal/response negotiation control

The local browser only focuses on two specific calls:

// 将我的会话描述告知我的浏览器
pc.setLocalDescription(mySessionDescription)
...
// 将对等端的会话描述告知我的浏览器
pc.setRemoteDescription(yourSessionDescription)

Generate proposals and responses:

// 生成提议
pc.createOffer(gotOffer, didntGetOffer)

function gotOffer(aSessionDescription) {
    setLocalDescription(aSessionDescription)
    ...
    // 现在可以将会话描述（提议offer）发送给对等端，以便对等端
    // a)、将提议传递给setRemoteDescription
    // b)、调用createAnswer
}

// 生成应答
pc.createAnswer(gotAnswer, didntGetAnswer)

function gotAnswer(aSessionDescription) {
    setLocalDescription(aSessionDescription)
    ...
    // 现在将会话描述（应答answer）发送给对等端，以便对等端
    // a)、将应答传递给setRemoteDescription
}

4. Test demo description

The following test demo shows real-time video calls in two browsers, source code acquisition: please join the group 973961276

Data channel

RTCDataChannel , the data channel is a non-media interactive connection established between browsers. That is, no media messages are transmitted, and data is passed directly by bypassing the server. Compared with WebSocket and http messages, the data channel supports large traffic and low latency.

注意：
    单个对等连接中的多个数据通道底层共享一个流，所以只需一次offer、answer即可建立首个数据通道。之后再建立数据通道无需再次进行offer、answer交换。
    
    典型应用：游戏实时状态更新。

Use of data channels

The data channel can be created only after the RTCPeerConnection instance is created, as follows:

pc = new RTCPeerConnection()
dc = pc.createDataChannel('')

After one end has created the data channel, the other end only needs to listen to the ondatachannel event:

pc = new RTCPeerConnection()
pc.ondatachannel = function(e) {
    dc = e.channel
}

At this point, the two peers have established data channels with each other and can send messages directly to each other:

dc.send('i am a text string for sending')
dc.send(new Blob(['i am a blob object'], {type: 'text/plain'}))
dc.send(new arrayBuffer(32))    // 发送arrayBuffer
dc.onmessage = function(e) {
    console.log('收到消息：', e.data)
}

Test demo after adding data channel

Project source code acquisition: please add group 973961276

Some screenshots: