(Translation + Note) WebRTC perfect negotiation mode

Food guide:

If you don’t know much about WebRTC, you can read it firstUnderstand Web real-time communication from 0 through front-end video;
The original text comes from MDN Establishing a connection: The WebRTC perfect negotiation pattern. This article was translated using Google and revised into easy-to-understand sentences. Coupled with my own understanding, it is inevitable that there will be mistakes. I hope friends can correct them in the comment area;
If it is difficult to find close words in Chinese, English will be retained. For example, A polite peer and A impolite peer are literally translated as 有礼貌的对等 and 不礼貌的对等. Obviously, it is difficult to understand what they are expressing. Therefore, there will be comments at the first place used, and the original text will be retained thereafter;

Glossary

Peers: peers are p2p, peer connections are p2p connections;
Signaling: To establish a p2p connection, the two parties need to exchange information. The exchange channel can be websocket, https request, or even datachannel channel. There is no clear limit on the signaling channel. It is generally a websocket connection. For simplicity, the code example in this article is directly Work on one page;
Perfect Negotiation: TODO:

Establishing a connection: WebRTC perfect negotiation mode

This article introduces WebRTCPerfect Negotiation, describing how it works and why it negotiates WebRTC connections between peers Recommended approach, and provides sample code to demonstrate the technique.

Because WebRTC does not mandate a specific signaling mechanism during the negotiation of new peer connections, it is very flexible. However, despite this flexibility in the transmission and communication of signaling messages, there is still a recommended design pattern that you should follow whenever possible, called perfect negotiation.

After browsers started supporting WebRTC, it was realized that some parts of the negotiation process were more complex than required for typical use cases. This is due to a small number of issues with the API and some potential race conditions that need to be prevented. These issues have been resolved, allowing us to greatly simplify WebRTC negotiation. Perfect Negotiation mode is an example of how negotiation has improved since the early days of WebRTC.

perfect negotiation concept

Perfect negotiation completely separates the negotiation process from the rest of the application logic, seamlessly. Negotiation is essentially an asymmetric operation: one party needs to act as the "caller" and the other party is the "callee". The Perfect Negotiation pattern eliminates this difference by separating it into independent negotiation logic, so your application doesn't need to care which end of the connection it is on. As far as your application is concerned, it makes no difference whether you make an outgoing call or receive a call.

The biggest advantage of perfect negotiation is that both caller and callee use the same code, so there is no need to write duplicate or otherwise add levels of negotiation code.

//Comment starts

The above mentioned a perfect negotiation but I still don’t know what to express. This actually requires understanding the p2p connection process and the changes in signaling status:

p2p connection:

Code version:

(async () => {
    const peer1 = new RTCPeerConnection();
    const peer2 = new RTCPeerConnection();
    const offer = await peer1.createOffer();
    await peer1.setLocalDescription(offer);
    await peer2.setRemoteDescription(offer);
    const answer = await peer2.createAnswer();
    await peer2.setLocalDescription(asnwer);
    peer1.setRemoteDescription(answer);
})();

Picture version:

The above process can be summarized simply as follows: the initiator creates an offer, sets it locally, and sends it to the other party. The other party chooses to set your offer, and then creates the corresponding answer. After setting it locally, it returns it to the initiator, and the initiator then sets the answer. Complete this process.

Initiator's signaling changes:have-local-offer -> stable Called party's signaling changes:have-remote-offer -> stable

Signaling status:

As you can see from the figure, there are many signaling states, and strict process control is required to finally reach the statestable

If both parties generate offers locally at the same time and then send them to the other party at the same time, the negotiation will definitely fail.

Therefore, perfect negotiation is a technical solution that can solve this problem.

// End of comment

Perfect negotiation works by assigning each of the two peers a role to play in the negotiation process (I didn’t expect there was role playing in the code too -_-!!), which role is completely separate from the WebRTC connection state:

A polite peer, if the signaling to be set conflicts with its own signaling status, it will use rollback and use its own After the signaling status changes to stable, continue to set the signaling sent by the other party;
A impolite peer, if the signaling to be set conflicts with its own signaling status, it will give up setting the other party's signaling ( Subject to me);

This way, both parties know what will happen if there is a conflict between sent offers. Responses to error conditions become more predictable.

There is no limit on who is the polite peer or the important peer on both sides and can be random.

Implement perfect negotiation (code implementation)

Let’s look at an example of implementing the perfect negotiation model. This code assumes thatSignalingChannel defines a class for communicating with the signaling server. Of course, your own code can use whatever signaling technique you like.

Note that this code is the same for both peers involved in the connection.

Create signaling and peering connections

First, the signaling channel needs to be opened, which requiresRTCPeerConnection to be created. The STUN server listed here is obviously not a real server; you need to replace it with the address of a real STUN server. stun.myserver.tld

const config = {
  iceServers: [{ urls: "stun:stun.mystunserver.tld" }]
};

const signaler = new SignalingChannel();
const pc = new RTCPeerConnection(config);

This code also<video>gets elements using classes "selfview" and "remoteview"; these will contain the local user's self respectively View and view of the incoming stream from the remote peer.

Connect to remote peer

const constraints = { audio: true, video: true };
const selfVideo = document.querySelector("video.selfview");
const remoteVideo = document.querySelector("video.remoteview");


async function start() {
  try {
    const stream = await navigator.mediaDevices.getUserMedia(constraints);

    for (const track of stream.getTracks()) {
      pc.addTrack(track, stream);
    }
    selfVideo.srcObject = stream;
  } catch(err) {
    console.error(err);
  }
}

start()The function shown above can be called by either of the two endpoints that want to talk to each other. It doesn’t matter who does it first, negotiation will work.

This is not significantly different from the old WebRTC connection establishment code. The user's camera and microphone are obtained by calling getUserMedia(). The resulting media tracksRTCPeerConnection are then passed toaddTrack(). Then, finally, <video> is shown by selfVideo its own camera and microphone stream elements, allowing local Users see what other peers see.

Handle received tracks

Next we need to set up a handler for thetrack event to handle the inbound video that has been negotiated to be received over this peer connection and audio tracks. To do this, we implemented the RTCPeerConnection's ontrack event handler.

pc.ontrack = ({track, streams}) => {
  track.onunmute = () => {
    if (remoteVideo.srcObject) {
      return;
    }
    remoteVideo.srcObject = streams[0];
  };
};

This handler executes when the track event occurs. Usedeconstruction to extractRTCTrackEvent's The track and streams properties. The former is the video track or audio track being received. The latter is an array ofMediaStream objects, each object representing a stream containing this track (in rare cases, a track may belong to more than one flow). In our case this will always contain a stream at index 0 because we previously passed it a streamaddTrack().

Perfect negotiation logic

Now we get to the truly perfect negotiation logic, which functions completely independently of the rest of the application.

Handle events required for negotiation

First, we implementRTCPeerConnectionevent handleronnegotiationneeded to get the local description and send it to the remote peer using the signaling channel.

let makingOffer = false;

pc.onnegotiationneeded = async () => {
  try {
    makingOffer = true;
    await pc.setLocalDescription();
    signaler.send({ description: pc.localDescription });
  } catch(err) {
    console.error(err);
  } finally {
    makingOffer = false;
  }
};

Note:onnegotiationneeded is some status change of the pc, and then the callback needs to be negotiated. It may be that a track or a datachannel is added to the pc

Please note that setLocalDescription() without parameters will be based on the current signalingState. The collection description is either a pair from the remote The answer to the peer's latest offer, or is the newly created offer (if no negotiation took place). Here, it will always be an offer because the event required for negotiation is only triggered in the stable state.

We set a Boolean variable, makingOffer to true mark that we are preparing an offer. To avoid races, we will later use this value instead of the signaling state to determine if the offer is being processed, since the valuesignalingState changes asynchronously, thus introducing razzle-dazzle Opportunity.

After creates, sets, and sends an offer (or an error occurs), makingOffer will be reset to false.

Handling ICE candidates

Next, we need to handleRTCPeerConnectionevent icecandidate, this is how the local ICE layer passes the candidates to us for delivery to the remote peer over the signaling channel

pc.onicecandidate = ({candidate}) => signaler.send({candidate});

This will getcandidate members of this ICE event and pass them to the signaling channel's send() method for sending to the remote via the signaling server Peer.

Handle signaling messages

The final piece of the puzzle is the code that handles incoming messages from the signaling server. This is implemented here as an event handler on theonmessagesignal channel object. This method is called every time a message arrives from the signaling server.

let ignoreOffer = false;

signaler.onmessage = async ({ data: { description, candidate } }) => {
  try {
    if (description) {
      // 如果对端发过来的描述类型为offer前提下，如果本地正在生成offer，或者本地的信令状态不为stable，就认为是信令冲突
      const offerCollision = (description.type == "offer") &&
                             (makingOffer || pc.signalingState != "stable");

      ignoreOffer = !polite && offerCollision;
      if (ignoreOffer) {
        // 如果是impolite方，并且信令冲突，那么不管三七二十一，直接不处理
        return;
      }

      await pc.setRemoteDescription(description);
      if (description.type == "offer") {
        await pc.setLocalDescription();
        signaler.send({ description: pc.localDescription })
      }
    } else if (candidate) {
      try {
        await pc.addIceCandidate(candidate);
      } catch(err) {
        if (!ignoreOffer) {
          throw err;
        }
      }
    }
  } catch(err) {
    console.error(err);
  }
}

After SignalingChannel receives an incoming message from via its onmessage event handler, the received JSON object will be deconstructed to obtain its description or candidate. If the incoming message has description, it is a proposal or reply sent by the other peer.

On the other hand, if the message hascandidate, it is an ICE candidate received from the remote peer astrickle ICE.addIceCandidate() (tricle belongs to candidate-related knowledge and will not be introduced too much here). The candidate is destined by passing it to

After receiving description (offer or answer)

If we receive description, we are ready to respond to the incoming offer or answer. First, we check to make sure we are in a position to accept the offer. If the signaling status of the connection is not stable or if our connection end starts its own offer process, then we need to pay attention offer conflict.

If we are an important peer and we receive a conflicting offer, we will return without setting the description and instead set ignoreOffer to trueTo ensure that we also ignore all candidates that the other party may send to us on the signaling channel belonging to thisoffer.

If we are a polite peer and we receive a conflicting offer, we do not need to do anything special as our existing offer will be automatically rolled back in the next step.

After ensuring that we want to accept the offer, we set the remote description to the incoming offer by calling setRemoteDescription(). This lets WebRTC know what the recommended configuration is for other peers. If we are a polite peer, we will abandon our proposal and accept a new proposal.

If the remote description of the new setting is an offer, we ask WebRTC to select the appropriate local configuration by calling the RTCPeerConnection method without parameters setLocalDescription(). This causessetLocalDescription() to automatically generate appropriate answers in response to received offers. We then send the answer back to the first peer via the signaling channel.

Improve the code

If you’re curious about what makes a perfect negotiation, this section is for you. Here we'll look at each change made to the WebRTC API along with best practice recommendations to make flawless negotiation possible.

Calling setLocalDescription() without parameters

In the past,negotiationneeded events were easily handled in a way that easily called setLocalDescription with arguments - that is, it happened easily Collisions, in which case both parties may end up trying to make offers at the same time, causing one or the other peer to receive an error and abort the connection attempt.

old way

// bad case
pc.onnegotiationneeded = async () => {
  try {
    await pc.setLocalDescription(await pc.createOffer());
    signaler.send({description: pc.localDescription});
  } catch(err) {
    console.error(err);
  }
};

Because thecreateOffer() method is asynchronous and takes some time to complete, the remote peer may be Trying to send our own offer, causing us to leave the stable state and enter the have-remote-offer state, which means we are now waiting for an offer. But once it receives the offer we just sent, so does the remote peer. This leaves both parties in a state where they cannot complete the connection attempt.

Use new calling method

As shown in the Achieving perfect negotiation section, we can do this by introducing a variable (here called makingOffer ) to eliminate this problem, we use it to indicate that we are sending an offer, and use the updated setLocalDescription() method:

// good case
let makingOffer = false;

pc.onnegotiationneeded = async () => {
  try {
    makingOffer = true;
    await pc.setLocalDescription();
    signaler.send({ description: pc.localDescription });
  } catch(err) {
    console.error(err);
  } finally {
    makingOffer = false;
  }
};

WemakingOfferset it immediately before callingsetLocalDescription() to prevent interference from sending the offer, and we do not clear it back,falseUntil the offer has been sent to the signaling server (or an error occurred, preventing the offer from being generated). In this way, we can avoid the risk of offer conflicts.

finally realized

let ignoreOffer = false;

signaler.onmessage = async ({ data: { description, candidate } }) => {
  try {
    if (description) {
      const offerCollision = (description.type == "offer") &&
                             (makingOffer || pc.signalingState != "stable");

      ignoreOffer = !polite && offerCollision;
      if (ignoreOffer) {
        return;
      }

      await pc.setRemoteDescription(description);
      if (description.type == "offer") {
        await pc.setLocalDescription();
        signaler.send({ description: pc.localDescription });
      }
    } else if (candidate) {
      try {
        await pc.addIceCandidate(candidate);
      } catch(err) {
        if (!ignoreOffer) {
          throw err;
        }
      }
    }
  } catch(err) {
    console.error(err);
  }
}

Original text (translation + annotation) WebRTC perfect negotiation mode - Nuggets

★The business card at the end of the article allows you to receive free audio and video development learning materials, including (FFmpeg, webRTC, rtmp, hls, rtsp, ffplay, srs) and audio and video learning roadmap, etc.

See below! ↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓