How WebRTC supports H265 on the web

Since H265 is affected by copyright and browsers are unwilling to support it, webrtc cannot implement H265 decoding and playback. However, h265 is needed to solve the bandwidth problem at work, so the browser does not support H265 playback. I learned from Du Niang that you can replace the rtp packet by sending sctp (webrtc's data channel can use three protocols: 1. SCTP; 2. RTP; 3. QUIC. Let's study the first one here). The web side receives the data, then decodes and displays it.

Currently Google supports:

WebCodecs(browserInfo.type.toLowerCase()===’chrome’ && browserInfo.version >= 107&&(location.protocol===’https:’||location)

MSE(isTypeSupported(‘video/mp4; codecs=”hev1.1.6.L123.b0″‘))

ffmpeg wasm, this article explains the use of ffmpeg wasm soft solution.

SCTP

The full name of SCTP is Stream Control Transmission Protocol. It is a transmission protocol that uses port 5000 by default. Its position in the TCP/IP protocol stack is similar to TCP and UDP, and it has the characteristics of both TCP and UDP.

sdpinteraction 

offer:

a=end-of-candidates
m=application 9 UDP/DTLS/SCTP webrtc-datachannel
c=IN IP4 23.101.8.213
a=sendrecv
a=sctp-port:5000
a=mid:data
a=ice-ufrag:ePgh
a=ice-pwd:ZURiB67/xs69E76aOa7JDw
a=ice-options:trickle
a=fingerprint:sha-256 EF:7A:50:9C:05:8C:EF:84:4D:72:B2:74:30:BA:FD:82:76:D1:C3:FE:0C:A0:10:43:B8:6C:B2:ED:B3:F7:77:8B
a=setup:actpass
a=candidate:1 1 udp 2013266431 23.101.8.213 41901 typ host
a=end-of-candidates

answer:

m=application 9 UDP/DTLS/SCTP webrtc-datachannel
c=IN IP4 0.0.0.0
a=sendrecv
a=ice-pwd:518cb4fc626f82bef2ada4e9221dfb50
a=ice-ufrag:d5484fed
a=mid:data
a=setup:active
a=sctp-port:5000
a=max-message-size:1073741823

static void  onapplication(void* param, uint32_t track,/* uint32_t object, */int width, int height, const void* extra, size_t bytes)
{
  ice_session_t* s = (ice_session_t*)param;
  // 6.2.  SDP Parameters
  static const char* pattern =
    "m=application 9 UDP/DTLS/SCTP webrtc-datachanneln"
    "a=mid:2n"
    "a=sctp-port:5000n"
    "a=sendrecvn";
      "a=setup:passiven";
  strcat((char*)s->app.sender.buffer, pattern);
  int n = snprintf(s->app.sdp, sizeof(s->app.sdp), "%s", (char*)s->app.sender.buffer);
  n += ice_transport_getsdp(s->avt, s->app.stream, (char*)s->app.sender.buffer + n, sizeof(s->app.sender.buffer) - n);
}

data transmission

DataChannel data is encrypted by boringssl and then sent to the peer through udp. The same socket is used to send datachannel data and audio and video data. Audio and video data transmission uses the RTP protocol, which uses NACK and FEC strategies to resist packet loss, while the bottom layer of DataChannel uses the SCTP (stream control transmission protocol) protocol. SCTP is similar to TCP and is a reliable protocol with flow control and congestion control. transmission protocol, so no additional measures are required to achieve reliable transmission of DataChannel data. webrtc uses the third-party open source library usrsctplib to implement the SCTP protocol.

Note: usrsctp provides UDP encapsulation of SCTP, but we need these messages to be encapsulated in DTLS and actually sent/received by the same DTLS socket we interacted with before, not by usrscp itself... Therefore, we use the AF_CONN method.

demo

int ice_transport_send_sctp(struct ice_transport_t* avt, int streamid, uint64_t time, int flags ,const void* data, int bytes)
{
  int n = RTP_FIXED_HEADER + bytes;
  unsigned char* ptr = (uint8_t*)calloc( 1,n+1);
  rtp_header_t header;
   memset(&header,0,sizeof(rtp_header_t));
  header.timestamp = 1667895211;
  header.v = RTP_VERSION;
  header.pt = 100;
  header.ssrc = bytes;
  nbo_write_rtp_header(ptr, &header);
  memcpy(ptr + RTP_FIXED_HEADER, data, bytes);//data 265的裸数据
  janus_dtls_wrap_sctp_data(avt->dtls, "doc-datachannel", "udp",0, ptr, n);
  free(ptr);//后面优化释放
  return -1;
}

webreceive

Receive data and parse the rtp header:

function ReadBig32(array, index) {
    return ((array[index] << 24)     |
            (array[index + 1] << 16) |
            (array[index + 2] << 8)  |
            (array[index + 3]));
}
function initH265DC(pc, player) {
    console.log("initH265DC", Date());
    bFindFirstKeyFrame = false;
    bRecH265 = false;
    isKeyFrame = false;
    receivet1 = new Date().getTime();
    h265DC = pc.createDataChannel("webrtc-datachannel");

    // var ctx = canvas.getContext("2d");
    console.log("initH265DC0", h265DC);
    h265DC.onmessage = function (event) {
       // console.log("receive message: ",event.data.slice(12));
        //console.log("receive message: ", buf2hex(event.data));
        let data = new Uint8Array(event.data);
        version = ((data[0] & 0xC0) >>> 6);
    padding = ((data[0] & 0x20) >>> 5);
    itemCount = (data[0] & 0x1F);
    packetType =  data[1];
        let pts = ReadBig32(data, 4);

        let len = ReadBig32(data, 8);
        console.log("initH265DC0", version,padding ,itemCount,packetType,pts,len);
        //console.log("receive message: ", buf2hex(event.data.split(12)));event.data.byteLength-12
        var req = {
            t: ksendPlayerVideoFrameReq,
            l: len,
            d:  data.subarray(12).buffer
        };
        player.postMessage(req, [req.d]);

    };

reference

• [1] github
• [2] flv-h265.js、webrtc_H265player.juans
• [3] C/C++ programming for WebAssembly