webrtc file transfer of database

1. A seemingly simple thing is often not simple

A simple thing will often pour your heart and soul into it. Maybe you see that it is very simple, but it is not that simple; in fact, when you think about a lot of code in your spare time this year, it seems that there are not many that can really be finished.

It's almost the end of the year, and I'm writing this article. Every line of program is like writing a novel, good or bad; the code is poured with my heart.

It is not an easy task to really make a thing good; to make it easy to use, to be used by many people, and to have good altruism, requires constant quantitative changes.

webrtc demo URL:

File Download Test

2. Why give up Baidu Netdisk

It took about half a year to store the data on the Baidu network disk, and later found some fatal problems

1. It is found that the shared external links cannot be shared after reaching 100,000 or tens of thousands

2. Baidu external links shared are often blocked

3. The sharing speed should not be too fast. You can only share more than 100 external links in one hour. If you share too fast, the sharing function of the entire network disk will be disabled.

4. Baidu account requires real-name authentication, and it is easy to be recognized by others. As a technician, I think it should be hidden behind the scenes, and try not to let others recognize it, so as to avoid many unnecessary problems this time

3. Why not use cloud storage

4 T storage costs 400 yuan per month, 5600 yuan a year, and 5600 yuan a year without traffic charges.

1. It may be totaled by hackers. If someone uses curl to maliciously brush traffic, the existing security policy may be stolen by someone.

2. The annual storage fee is increasing

3. Real-name authentication is required. Once the domain name is bound and the mobile phone number is bound, it is likely to be found

4. My WebRtc architecture design

Please rely on the traditional relay server, use p2p to drill holes, research some solutions, originally planned to use coturn + janus architecture design, but found that janus c wrote, there is no file transfer plug-in, if you use c to make a plug-in really It is very tiring, so I plan to use golang to implement a file transfer gateway.

general steps

1. Start the gateway

2. The storage node is connected to the gateway

3. The user browser requests the gateway, and the request exchanges signaling with the storage node

4. The signaling exchange is completed, and dataChannel is used for communication

5. Gateway design

The gateway is mainly a websocket server written in golang

The gateway design is mainly divided into several modules

1、room

room is a public domain, mainly for client and node authentication operations, if the authentication is successful, it will enter the Manager for scheduling

2. Abstract client and node

The commonality between client and node is that they both have connection attributes, so a common interface should be designed to abstract their commonality

type Lifecycle interface {
	Stop()
	Start()
}

type Describable interface {
	Name() string
	Type() ConnType
}


type Component interface {
	Lifecycle
	Describable
	AddCallback(ctx EventCallback)
	GetConnNumber() uint64
	SetContext(interface{})
	GetContext() interface{}
	Product(ctx Context)
}

By adding an AddCallback callback function, it is ensured that the business processing of different modules is completely opened. The logic of the node is processed in the node, and the logic of the client is only processed in the client. The codes of different modules cannot be cross-processed. As for the transmission of up and down questions, Unified abstraction of a Context, which abstractly stores the context information we need, for different callback functions and transmission between collaborations.

type Context interface {
	GetData() []byte
	Error() error
	MessageType() int
	GetContext() interface{}
	SetContext(interface{})
	Name() string
}

type EventCallback interface {
	OnReceive(Context, *sync.WaitGroup)
	OnError(Context, *sync.WaitGroup)
	OnClose(Context, *sync.WaitGroup)
	OnWriteComplete(Context, *sync.WaitGroup)
}

type NodeCallback interface {
	OnReceive(Context, *sync.WaitGroup)
	OnError(Context, *sync.WaitGroup)
	OnClose(Context, *sync.WaitGroup)
	OnWriteComplete(Context, *sync.WaitGroup)
}

4、manager

Responsible for overall scheduling. For example, after the client enters the manager, it searches for currently available node storage nodes, and performs signaling exchange after finding them.

Node scheduling, traversal to find the node node with the least client, and then communicate

// 选择最优线路
func (m *Manager) selectNode() *node.NodeClient {
	if len(m.nodeTree) == 0 {
		return nil
	}

	// 找一个挂载链接最少的节点
	var usableNode *node.NodeClient
	var weight uint64
	for _, conn := range m.nodeTree {
		if uint64(conn.GetConnNumber()) <= weight {
			usableNode = conn.(*node.NodeClient)
		}
	}
	return usableNode
}

6. The node node responds after receiving the signaling

1. Create RTCConnection

blockSize := 16384
	//前端页面会对sdp进行base64的encode
	b, err := base64.StdEncoding.DecodeString(sdp)
	if err != nil {
		log.Error("error:%s", err)
		return nil
	}

	str, err := url.QueryUnescape(string(b))
	if err != nil {
		log.Error("error:%s", err)
		return nil
	}

	sdpDes := webrtc.SessionDescription{}
	fmt.Println(str)
	err = json.Unmarshal([]byte(str), &sdpDes)
	if err != nil {
		log.Error("json.Unmarshal err:%s", err)
		return nil
	}

	//创建pc， 并且指定stun服务器
	pc, err := webrtc.NewPeerConnection(webrtc.Configuration{
		ICEServers: []webrtc.ICEServer{
			{
				URLs: []string{"stun:"},
			},
		},
	})

	stat, err := os.Stat("/home/zhanglei/Downloads/《实践论》(原文)毛泽东.pdf")
	if err != nil {
		log.Error("os.Stat %s error:%s", path, err)
		return nil
	}

	if offset > stat.Size() {
		log.Error("offset(%d) > stat.Size(%d)", offset, stat.Size())
		return nil
	}

	chunkSize := int(math.Ceil(float64(stat.Size() / int64(blockSize))))
	currentChunkSize := int(math.Ceil(float64(offset / int64(blockSize))))

	if err != nil {
		log.Error("%s", err)
		return nil
	}

	pc.OnConnectionStateChange(func(s webrtc.PeerConnectionState) {
		fmt.Println("OnConnectionStateChange")
		fmt.Printf("Peer Connection State has changed: %s (answerer)\n", s.String())
		if s == webrtc.PeerConnectionStateFailed {
			fmt.Println("webrtc.PeerConnectionStateFailed")
		}
	})

	// Register data channel creation handling
	pc.OnDataChannel(func(d *webrtc.DataChannel) {
		fmt.Printf("New DataChannel %s %d\n", d.Label(), d.ID())

		// Register channel opening handling
		d.OnOpen(func() {
			fmt.Printf("Data channel '%s'-'%d' open. Random messages will now be sent to any connected DataChannels every 5 seconds\n", d.Label(), d.ID())

			stat, err := os.Stat("/home/zhanglei/Downloads/《实践论》(原文)毛泽东.pdf")
			chunkSize = int(math.Ceil(float64(stat.Size() / int64(blockSize))))

			// 握手
			var chunk ChunkMessage
			chunk.Class = HANDSHAKE
			chunk.ChunkSize = uint64(chunkSize)
			handShakeBytes := serialize(&chunk)

			err = d.Send(handShakeBytes.Bytes())
			if err != nil {
				log.Error("%s", err)
				return
			}
		})

		// Register text message handling
		d.OnMessage(func(msg webrtc.DataChannelMessage) {
			fmt.Printf("Message from DataChannel '%s': '%s'\n", d.Label(), string(msg.Data))
			data, err := unSerialize(msg.Data)
			if err != nil {
				log.Error("os.Open is : %s", err)
				return
			}

			if data.Class == ACK {
				handle, err := os.Open("/home/zhanglei/Downloads/《实践论》(原文)毛泽东.pdf")
				if err != nil {
					log.Error("os.Open is : %s", err)
					return
				}

				defer handle.Close()

				bufferBytes := make([]byte, blockSize)
				read, err := handle.Read(bufferBytes)
				if err != nil {
					log.Error("handle.Read is : %s", err)
					return
				}

				if read < blockSize {
					bufferBytes = bufferBytes[:read]
				}

				var chunk ChunkMessage
				chunk.Class = SEND
				chunk.ChunkSize = uint64(chunkSize)
				chunk.CurrentChunk = uint64(currentChunkSize)
				chunk.Data = bufferBytes

				// 打包发送
				err = d.Send(serialize(&chunk).Bytes())
				if err != nil {
					log.Error("%s", err)
					return
				}
				return
			}

			if data.Class == RECEIVE {
				handle, err := os.Open("/home/zhanglei/Downloads/《实践论》(原文)毛泽东.pdf")
				if err != nil {
					log.Error("os.Open is : %s", err)
					return
				}

				if data.CurrentChunk == uint64(chunkSize) {
					log.Info("data transfer finish")
					return
				}

				nextChunk := data.CurrentChunk + 1

				bytes := make([]byte, blockSize)
				read, err := handle.ReadAt(bytes, int64(nextChunk)*int64(blockSize))

				if err != nil {
					if !errors.Is(err, io.EOF) {
						log.Error("handle.Read is : %s", err)
						return
					}

				}

				if read < blockSize {
					bytes = bytes[:read]
				}

				var sendData ChunkMessage
				sendData.Class = SEND
				sendData.CurrentChunk = nextChunk
				sendData.Data = bytes
				sendData.ChunkSize = uint64(chunkSize)
				sendDataBytes := serialize(&sendData)
				log.Info(" read %d", nextChunk)

				err = d.Send(sendDataBytes.Bytes())
				if err != nil {
					log.Error("%s", err)
					return
				}

				// 最后一块
				if nextChunk == uint64(chunkSize) {
					d.Close()
					pc.Close()
				}
				return
			}

		})
	})

	_, err = pc.CreateDataChannel("sendDataChannel", nil)
	if err != nil {
		log.Error("error:%s", err)
		return nil
	}

	//channel.OnOpen(func() {
	//
	//})

	//设置远端的sdp
	if err = pc.SetRemoteDescription(sdpDes); err != nil {
		log.Error("error:%s", err)
		return nil
	}

	//创建协商结果
	answer, err := pc.CreateAnswer(nil)
	if err != nil {
		log.Error("error:%s", err)
		return nil
	}

	pc.OnICECandidate(func(i *webrtc.ICECandidate) {
		fmt.Println("OnICECandidate")
		fmt.Println(i)

	})

	err = pc.SetLocalDescription(answer)
	if err != nil {
		log.Error("error:%s", err)
		return nil
	}

	//等待ice结束
	gatherCmp := webrtc.GatheringCompletePromise(pc)
	<-gatherCmp

	//将协商并且收集完candidate的answer，输出到控制台
	answerBytes, err := json.Marshal(*pc.LocalDescription())
	if err != nil {
		log.Error("error:%s", err)
		return nil
	}

	pc.OnICECandidate(func(candidate *webrtc.ICECandidate) {
		fmt.Println("OnICECandidate")
		fmt.Println(candidate)
	})

	t := &Transfer{
		sdp:              sdp,
		pc:               pc,
		offset:           uint64(offset),
		currentChunkSize: currentChunkSize,
		chunkSize:        chunkSize,
		answerSdp:        answerBytes,
		blockSize:        blockSize,
	}
	return t

2. Package transmission

The transmission did not use protobuf, and I wrote a binary transmission myself

func serialize(data *ChunkMessage) *bytes.Buffer {
	data.Version = CodeVersion
	writeBuffer := bytes.NewBuffer(nil)
	writeBuffer.Write([]byte{data.Version})
	writeBuffer.Write([]byte{data.Class})

	// ChunkSize
	binary.Write(writeBuffer, binary.BigEndian, data.ChunkSize)

	// CurrentChunk
	binary.Write(writeBuffer, binary.BigEndian, data.CurrentChunk)

	// DataLen
	data.DataLen = uint64(len(data.Data))
	binary.Write(writeBuffer, binary.BigEndian, data.DataLen)

	if len(data.Data) > 0 {
		// 添加body
		writeBuffer.Write(data.Data)
	}

	return writeBuffer
}

//判断我们系统中的字节序类型
func systemEdian() binary.ByteOrder {
	var i int = 0x1
	bs := (*[int(unsafe.Sizeof(0))]byte)(unsafe.Pointer(&i))
	if bs[0] == 0 {
		return binary.LittleEndian
	} else {
		return binary.BigEndian
	}
}

func unSerialize(data []byte) (*ChunkMessage, error) {
	buf := bytes.NewBuffer(data)
	fmt.Println(buf)
	var chunk ChunkMessage
	binary.Read(buf, systemEdian(), &chunk.Version)
	binary.Read(buf, systemEdian(), &chunk.Class)
	binary.Read(buf, systemEdian(), &chunk.ChunkSize)
	binary.Read(buf, systemEdian(), &chunk.CurrentChunk)
	binary.Read(buf, systemEdian(), &chunk.DataLen)
	//chunkSize := uint64(unsafe.Pointer(&buf.Bytes()))
	//chunk.ChunkSize = chunkSize
	return &chunk, nil
}

7. The js front-end webrtc provides quotations

Create a webrtc connection

var pcConfig = {
            'iceServers': [{
                'urls': 'stun:',
            }]
        };
        localConnection = new RTCPeerConnection(pcConfig);

        receiveDataChannel = localConnection.createDataChannel("receiveDataChannel")

        receiveDataChannel.binaryType = "arraybuffer"

        receiveDataChannel.addEventListener('open', dataChannel.onopen);
        receiveDataChannel.addEventListener('close', dataChannel.onclose);
        receiveDataChannel.addEventListener('message', dataChannel.onmessage);
        receiveDataChannel.addEventListener('error', dataChannel.onError);

        try {
            this.offer = await localConnection.createOffer();
        } catch (e) {
            console.log('Failed to create session description: ', e);
            return
        }

        try {
            await localConnection.setLocalDescription(this.offer)
        } catch (e) {
            console.log('Failed to create session description: ', e);
            return
        }

        //eyJ1c2VyX3V1aWQiOiI1ZmZkNDE0N2JkMTMyNWNmMjYwNDAyMWYwODA5OWUyMyIsImxvZ2luX3RpbWUiOjE2Njg0NzgzOTEsIm5vd190aW1lIjoxNjY5OTgzNzkwLCJyYW5fc3RyIjoiZDA3MTczNzI3NjFjMzY0MGU2NmRlYWI5YmYyODZhNzYiLCJzaWduIjoiZjc3NzI0YjZmMTc3MzczNmVhZWFkMTM2NzllNTE0NTcifQ==

        transfer.ws.send((JSON.stringify(downloadRequest)));

The front end serializes and deserializes the received data

function serialize(data) {
    var bufLen = protoColMinSize;
    if (!data.Data) {
        bufLen += 0;
    } else {
        bufLen += data.Data.length;
    }
    data.Version = 1;
    var protocolBuf = new ArrayBuffer(bufLen);
    const bufView = new DataView(protocolBuf);
    bufView.setUint8(0, data.Version);

    bufView.setUint8(1, data.Class);

    if (!data.ChunkSize) {
        data.ChunkSize = 0
    }
    bufView.setBigUint64(2, BigInt(data.ChunkSize));


    if (!data.CurrentChunk) {
        data.CurrentChunk = 0
    }
    bufView.setBigUint64(10, BigInt(data.CurrentChunk));

    if (data.Data && data.Data.length > 0) {
        bufView.setBigUint64(18, BigInt(data.Data.length));
    } else {
        bufView.setBigUint64(18, BigInt(0));
    }

    console.log(protocolBuf)
    return protocolBuf;
}

function unSerialize(bytes) {

    var versionView = new DataView(bytes).getUint8(0);
    // 最小长度
    var classByteView = new DataView(bytes).getUint8(1);
    // chunk 长度
    var chunkSizeView = parseInt(new DataView(bytes).getBigUint64(2));
    var currentChunkView = parseInt(new DataView(bytes).getBigUint64(10));
    var bodyLenView = parseInt(new DataView(bytes).getBigUint64(18));
    var returnData = {
        Version: versionView,
        Class: classByteView,
        ChunkSize: (chunkSizeView),
        CurrentChunk: currentChunkView,
        PayloadLength: bodyLenView,
        Payload: [],
    };

    if (bodyLenView > 0) {
        returnData.Payload = new Uint8Array(bytes, protoColMinSize, bodyLenView)
    }

    return returnData;
}

8. Advantages and disadvantages of webrtc file transfer

1) Advantages

1. Point-to-point transmission without relay server

2. Civilian bandwidth is relatively cheap. In the worst case, the bandwidth is full, and there will be no high traffic charges

3. You can build your own storage, you can use Qunhui for storage nodes, and you can buy your own server

4. No need for a fixed ip address

2) Disadvantages

1. I don’t know how many broadband lines can be applied for for civilian use

2. Storage and maintenance of hardware is also a troublesome thing. The hard disk is likely to fail, and operation and maintenance is also a headache.