There are three theoretical solutions for multi-person video, as shown in the figure below, from left to right are Mesh, SFU, and MCU.
In a Mesh grid, each person establishes a separate connection with each other. In the case of 4 people, each person establishes 3 connections, that is, 3 upload streams and 3 download streams. This solution requires the highest client network and computing power , there is no special requirement for the server.
SFU (Selective Forwarding Unit) is an optional forwarding unit. There is a central unit responsible for forwarding streams. Each person only establishes a connection with the central unit, uploads their own streams, and downloads other people’s streams. In the case of 4 people, each person establishes One connection, including 1 upload stream and 3 download streams. This solution has higher requirements for the client and higher requirements for the server.
MCU (Multipoint Control Unit) is a multi-terminal control unit with a central unit responsible for mixing stream processing and forwarding streams. Each person only establishes a connection with the central unit, uploads their own stream, and downloads the mixed stream. In the case of 4 people, each person establishes A connection, including 1 upload stream and 1 download stream. This solution has no special requirements for the client, but has the highest requirements for the server.
Mesh implementation
First analyze theoretically, the connection between client A and B is completely through the PeerConnection object, so as long as client A has multiple PeerConnection objects, it can connect with B, C, D... at the same time.
Although there are multiple PeerConnections, client A is still connected to the signaling server through a socket, so when A sends a signal to the server, he must specify who to send it to, and when receiving the signal, he must determine who it is from, and the server receives the signal. It is necessary to determine who is sending the order. This requires adding two fields from and to to all signaling , representing the sender and receiver of the signaling. Each socket connection has a unique socketId, which can be used to identify a client . Each client uses a HashMap<String, PeerConnection> (key is socketId) to save its connection.
Dial plan: client A joins the room, if there are other clients B and C in the room, the server sends A’s socketId to B and C, B and C each send Offer to A to establish a connection after receiving it, and A replies to Answer respectively Passively establish multiple connections. This ensures that the logic of each client is the same. If it newly joins the room, it only needs to wait for other people's Offer; if it is already in the room, it waits for others to join and send Offer to others .
Signaling server
Make the following modifications on the basis of the previous article,
- When forwarding a message , select the sending target according to the to in it
- When someone joins the room, send the person's socketId to others
- Remove the limit of two people in a room
socket.on('message', function(message) {
// for a real app, would be room-only (not broadcast)
// socket.broadcast.emit('message', message);
var to = message['to'];
log('from:' + socket.id + " to:" + to, message);
io.sockets.sockets[to].emit('message', message);
});
socket.on('create or join', function(room) {
log('Received request to create or join room ' + room);
var clientsInRoom = io.sockets.adapter.rooms[room];
var numClients = clientsInRoom ? Object.keys(clientsInRoom.sockets).length : 0;
log('Room ' + room + ' now has ' + numClients + ' client(s)');
if (numClients === 0) {
socket.join(room);
log('Client ID ' + socket.id + ' created room ' + room);
socket.emit('created', room, socket.id);
} else {
log('Client ID ' + socket.id + ' joined room ' + room);
io.sockets.in(room).emit('join', room, socket.id);
socket.join(room);
socket.emit('joined', room, socket.id);
io.sockets.in(room).emit('ready');
}
});
Based on the previous article, MainActivity.java adds HashMap<String, PeerConnection> peerConnectionMap(key is socketId) to manage all PeerConnection connections, judges the socketId of the source when receiving the signal, and adds the socketId of itself and the other party when sending.
public class MainActivity extends AppCompatActivity implements SignalingClient.Callback {
EglBase.Context eglBaseContext;
PeerConnectionFactory peerConnectionFactory;
SurfaceViewRenderer localView;
MediaStream mediaStream;
List<PeerConnection.IceServer> iceServers;
HashMap<String, PeerConnection> peerConnectionMap;
SurfaceViewRenderer[] remoteViews;
int remoteViewsIndex = 0;
@Override
protected void onCreate(Bundle savedInstanceState) {
super.onCreate(savedInstanceState);
setContentView(R.layout.activity_main);
peerConnectionMap = new HashMap<>();
iceServers = new ArrayList<>();
iceServers.add(PeerConnection.IceServer.builder("stun:stun.l.google.com:19302").createIceServer());
eglBaseContext = EglBase.create().getEglBaseContext();
// create PeerConnectionFactory
PeerConnectionFactory.initialize(PeerConnectionFactory.InitializationOptions
.builder(this)
.createInitializationOptions());
PeerConnectionFactory.Options options = new PeerConnectionFactory.Options();
DefaultVideoEncoderFactory defaultVideoEncoderFactory =
new DefaultVideoEncoderFactory(eglBaseContext, true, true);
DefaultVideoDecoderFactory defaultVideoDecoderFactory =
new DefaultVideoDecoderFactory(eglBaseContext);
peerConnectionFactory = PeerConnectionFactory.builder()
.setOptions(options)
.setVideoEncoderFactory(defaultVideoEncoderFactory)
.setVideoDecoderFactory(defaultVideoDecoderFactory)
.createPeerConnectionFactory();
SurfaceTextureHelper surfaceTextureHelper = SurfaceTextureHelper.create("CaptureThread", eglBaseContext);
// create VideoCapturer
VideoCapturer videoCapturer = createCameraCapturer(true);
VideoSource videoSource = peerConnectionFactory.createVideoSource(videoCapturer.isScreencast());
videoCapturer.initialize(surfaceTextureHelper, getApplicationContext(), videoSource.getCapturerObserver());
videoCapturer.startCapture(480, 640, 30);
localView = findViewById(R.id.localView);
localView.setMirror(true);
localView.init(eglBaseContext, null);
// create VideoTrack
VideoTrack videoTrack = peerConnectionFactory.createVideoTrack("100", videoSource);
// // display in localView
videoTrack.addSink(localView);
remoteViews = new SurfaceViewRenderer[]{
findViewById(R.id.remoteView),
findViewById(R.id.remoteView2),
findViewById(R.id.remoteView3),
};
for(SurfaceViewRenderer remoteView : remoteViews) {
remoteView.setMirror(false);
remoteView.init(eglBaseContext, null);
}
mediaStream = peerConnectionFactory.createLocalMediaStream("mediaStream");
mediaStream.addTrack(videoTrack);
SignalingClient.get().init(this);
}
private synchronized PeerConnection getOrCreatePeerConnection(String socketId) {
PeerConnection peerConnection = peerConnectionMap.get(socketId);
if(peerConnection != null) {
return peerConnection;
}
peerConnection = peerConnectionFactory.createPeerConnection(iceServers, new PeerConnectionAdapter("PC:" + socketId) {
@Override
public void onIceCandidate(IceCandidate iceCandidate) {
super.onIceCandidate(iceCandidate);
SignalingClient.get().sendIceCandidate(iceCandidate, socketId);
}
@Override
public void onAddStream(MediaStream mediaStream) {
super.onAddStream(mediaStream);
VideoTrack remoteVideoTrack = mediaStream.videoTracks.get(0);
runOnUiThread(() -> {
remoteVideoTrack.addSink(remoteViews[remoteViewsIndex++]);
});
}
});
peerConnection.addStream(mediaStream);
peerConnectionMap.put(socketId, peerConnection);
return peerConnection;
}
@Override
public void onCreateRoom() {
}
@Override
public void onPeerJoined(String socketId) {
PeerConnection peerConnection = getOrCreatePeerConnection(socketId);
peerConnection.createOffer(new SdpAdapter("createOfferSdp:" + socketId) {
@Override
public void onCreateSuccess(SessionDescription sessionDescription) {
super.onCreateSuccess(sessionDescription);
peerConnection.setLocalDescription(new SdpAdapter("setLocalSdp:" + socketId), sessionDescription);
SignalingClient.get().sendSessionDescription(sessionDescription, socketId);
}
}, new MediaConstraints());
}
@Override
public void onSelfJoined() {
}
@Override
public void onPeerLeave(String msg) {
}
@Override
public void onOfferReceived(JSONObject data) {
runOnUiThread(() -> {
final String socketId = data.optString("from");
PeerConnection peerConnection = getOrCreatePeerConnection(socketId);
peerConnection.setRemoteDescription(new SdpAdapter("setRemoteSdp:" + socketId),
new SessionDescription(SessionDescription.Type.OFFER, data.optString("sdp")));
peerConnection.createAnswer(new SdpAdapter("localAnswerSdp") {
@Override
public void onCreateSuccess(SessionDescription sdp) {
super.onCreateSuccess(sdp);
peerConnectionMap.get(socketId).setLocalDescription(new SdpAdapter("setLocalSdp:" + socketId), sdp);
SignalingClient.get().sendSessionDescription(sdp, socketId);
}
}, new MediaConstraints());
});
}
@Override
public void onAnswerReceived(JSONObject data) {
String socketId = data.optString("from");
PeerConnection peerConnection = getOrCreatePeerConnection(socketId);
peerConnection.setRemoteDescription(new SdpAdapter("setRemoteSdp:" + socketId),
new SessionDescription(SessionDescription.Type.ANSWER, data.optString("sdp")));
}
@Override
public void onIceCandidateReceived(JSONObject data) {
String socketId = data.optString("from");
PeerConnection peerConnection = getOrCreatePeerConnection(socketId);
peerConnection.addIceCandidate(new IceCandidate(
data.optString("id"),
data.optInt("label"),
data.optString("candidate")
));
}
@Override
protected void onDestroy() {
super.onDestroy();
SignalingClient.get().destroy();
}
private VideoCapturer createCameraCapturer(boolean isFront) {
Camera1Enumerator enumerator = new Camera1Enumerator(false);
final String[] deviceNames = enumerator.getDeviceNames();
// First, try to find front facing camera
for (String deviceName : deviceNames) {
if (isFront ? enumerator.isFrontFacing(deviceName) : enumerator.isBackFacing(deviceName)) {
VideoCapturer videoCapturer = enumerator.createCapturer(deviceName, null);
if (videoCapturer != null) {
return videoCapturer;
}
}
}
return null;
}
}
multiplayer video
Start the node.js server, install the clients on multiple Android phones, start them one after another, and then you can see the images of everyone else on one client. (The layout file here only puts 4 SurfaceViewRenderers, so it supports 2, 3,4 mobile phones connected at the same time).
GitHub address of this project/step4multipeers
GitHub address of this project/step4web
Reprint: https://www.jianshu.com/p/8c10146afd6c