In the previous article, the most basic use of WebRTC - the use of the camera was completed. This article will introduce the core concept PeerConnection in WebRTC, which establishes a virtual connection for the front and rear cameras in the same mobile phone, and transmits images to each other.
PeerConnection
PeerConnection is also Peer-to-Peer connection (P2P), which is the connection between two "people". Both parties create PeerConnection objects respectively, and then send their own network status ICE and multimedia encoding format SDP to the other party (because the connection has not yet been established at this time , so the sending content is done through the server). When the network and encoding format of the two parties are negotiated, the connection is established. At this time, the MediaStream data stream of the other party can be obtained from the PeerConnection, and the audio and video of the other party can be played .
ICE
Interactive Connectivity Establishment, interactive connection establishment. In fact, it is a framework that integrates STUN and TURN. Provide STUN and TURN server addresses to it, and it will automatically select the one with higher priority for NAT penetration.
SDP
Session Description Protocol: Session Description Protocol. The sender is called Offer, and the receiver is called Answer. There is no difference except the name. It is some text describing the local audio and video encoding and network address.
main process
A (local) and B (remote) represent two people, initialize PeerConnectionFactory and create PeerConnection respectively, and add local media stream to PeerConnection.
- A Create an Offer
- A Save Offer (set local description)
- A sends Offer to B
- B Save Offer (set remote description)
- B Create Answer
- BSaveAnswer(set local description)
- B sends Answer to A
- A Save Answer (set remote description)
- A sends Ice Candidates to B
- B sends Ice Candidates to A
-
A and B receive each other's media stream and play it
As shown in the figure above, there are a total of 11 steps. Although there are many steps, they are not complicated. The two sides are basically symmetrical. The main code is as follows.
preparation steps
It is mainly to initialize PeerConnectionFactory and use the camera, which has been introduced in the previous article.
public class MainActivity extends AppCompatActivity {
PeerConnectionFactory peerConnectionFactory;
PeerConnection peerConnectionLocal;
PeerConnection peerConnectionRemote;
SurfaceViewRenderer localView;
SurfaceViewRenderer remoteView;
MediaStream mediaStreamLocal;
MediaStream mediaStreamRemote;
@Override
protected void onCreate(Bundle savedInstanceState) {
super.onCreate(savedInstanceState);
setContentView(R.layout.activity_main);
EglBase.Context eglBaseContext = EglBase.create().getEglBaseContext();
// create PeerConnectionFactory
PeerConnectionFactory.initialize(PeerConnectionFactory.InitializationOptions
.builder(this)
.createInitializationOptions());
PeerConnectionFactory.Options options = new PeerConnectionFactory.Options();
DefaultVideoEncoderFactory defaultVideoEncoderFactory =
new DefaultVideoEncoderFactory(eglBaseContext, true, true);
DefaultVideoDecoderFactory defaultVideoDecoderFactory =
new DefaultVideoDecoderFactory(eglBaseContext);
peerConnectionFactory = PeerConnectionFactory.builder()
.setOptions(options)
.setVideoEncoderFactory(defaultVideoEncoderFactory)
.setVideoDecoderFactory(defaultVideoDecoderFactory)
.createPeerConnectionFactory();
SurfaceTextureHelper surfaceTextureHelper = SurfaceTextureHelper.create("CaptureThread", eglBaseContext);
// create VideoCapturer
VideoCapturer videoCapturer = createCameraCapturer(true);
VideoSource videoSource = peerConnectionFactory.createVideoSource(videoCapturer.isScreencast());
videoCapturer.initialize(surfaceTextureHelper, getApplicationContext(), videoSource.getCapturerObserver());
videoCapturer.startCapture(480, 640, 30);
localView = findViewById(R.id.localView);
localView.setMirror(true);
localView.init(eglBaseContext, null);
// create VideoTrack
VideoTrack videoTrack = peerConnectionFactory.createVideoTrack("100", videoSource);
// // display in localView
// videoTrack.addSink(localView);
SurfaceTextureHelper remoteSurfaceTextureHelper = SurfaceTextureHelper.create("RemoteCaptureThread", eglBaseContext);
// create VideoCapturer
VideoCapturer remoteVideoCapturer = createCameraCapturer(false);
VideoSource remoteVideoSource = peerConnectionFactory.createVideoSource(remoteVideoCapturer.isScreencast());
remoteVideoCapturer.initialize(remoteSurfaceTextureHelper, getApplicationContext(), remoteVideoSource.getCapturerObserver());
remoteVideoCapturer.startCapture(480, 640, 30);
remoteView = findViewById(R.id.remoteView);
remoteView.setMirror(false);
remoteView.init(eglBaseContext, null);
// create VideoTrack
VideoTrack remoteVideoTrack = peerConnectionFactory.createVideoTrack("102", remoteVideoSource);
// // display in remoteView
// remoteVideoTrack.addSink(remoteView);
mediaStreamLocal = peerConnectionFactory.createLocalMediaStream("mediaStreamLocal");
mediaStreamLocal.addTrack(videoTrack);
mediaStreamRemote = peerConnectionFactory.createLocalMediaStream("mediaStreamRemote");
mediaStreamRemote.addTrack(remoteVideoTrack);
call(mediaStreamLocal, mediaStreamRemote);
}
}
use the camera
Make a slight modification to the createCameraCapturer() method, and pass in the boolean parameter to obtain the front and rear cameras respectively.
private VideoCapturer createCameraCapturer(boolean isFront) {
Camera1Enumerator enumerator = new Camera1Enumerator(false);
final String[] deviceNames = enumerator.getDeviceNames();
// First, try to find front facing camera
for (String deviceName : deviceNames) {
if (isFront ? enumerator.isFrontFacing(deviceName) : enumerator.isBackFacing(deviceName)) {
VideoCapturer videoCapturer = enumerator.createCapturer(deviceName, null);
if (videoCapturer != null) {
return videoCapturer;
}
}
}
return null;
}
dial
One of the two people who establish a connection must be the caller and the other is the receiver. The caller creates an Offer and sends it to the receiver, and the receiver replies with an Answer after receiving it.
private void call(MediaStream localMediaStream, MediaStream remoteMediaStream) {
List<PeerConnection.IceServer> iceServers = new ArrayList<>();
peerConnectionLocal = peerConnectionFactory.createPeerConnection(iceServers, new PeerConnectionAdapter("localconnection") {
@Override
public void onIceCandidate(IceCandidate iceCandidate) {
super.onIceCandidate(iceCandidate);
peerConnectionRemote.addIceCandidate(iceCandidate);
}
@Override
public void onAddStream(MediaStream mediaStream) {
super.onAddStream(mediaStream);
VideoTrack remoteVideoTrack = mediaStream.videoTracks.get(0);
runOnUiThread(() -> {
remoteVideoTrack.addSink(localView);
});
}
});
peerConnectionRemote = peerConnectionFactory.createPeerConnection(iceServers, new PeerConnectionAdapter("remoteconnection") {
@Override
public void onIceCandidate(IceCandidate iceCandidate) {
super.onIceCandidate(iceCandidate);
peerConnectionLocal.addIceCandidate(iceCandidate);
}
@Override
public void onAddStream(MediaStream mediaStream) {
super.onAddStream(mediaStream);
VideoTrack localVideoTrack = mediaStream.videoTracks.get(0);
runOnUiThread(() -> {
localVideoTrack.addSink(remoteView);
});
}
});
peerConnectionLocal.addStream(localMediaStream);
peerConnectionLocal.createOffer(new SdpAdapter("local offer sdp") {
@Override
public void onCreateSuccess(SessionDescription sessionDescription) {
super.onCreateSuccess(sessionDescription);
// todo crashed here
peerConnectionLocal.setLocalDescription(new SdpAdapter("local set local"), sessionDescription);
peerConnectionRemote.addStream(remoteMediaStream);
peerConnectionRemote.setRemoteDescription(new SdpAdapter("remote set remote"), sessionDescription);
peerConnectionRemote.createAnswer(new SdpAdapter("remote answer sdp") {
@Override
public void onCreateSuccess(SessionDescription sdp) {
super.onCreateSuccess(sdp);
peerConnectionRemote.setLocalDescription(new SdpAdapter("remote set local"), sdp);
peerConnectionLocal.setRemoteDescription(new SdpAdapter("local set remote"), sdp);
}
}, new MediaConstraints());
}
}, new MediaConstraints());
}
}
Note: Although the network is not actually used here, it is necessary to add network permissions
<uses-permission android:name="android.permission.INTERNET"/>
<uses-permission android:name="android.permission.ACCESS_NETWORK_STATE"/>
Most of the local loopback (Loopback) demos on the Internet only use one camera. Here, the front and rear cameras of the same mobile phone are used, and they are used as two clients to establish a simulated connection and send media data. This is very similar to the actual WebRTC workflow. It's close, there is only one difference - the data transfer here is memory sharing, but it is actually sent over the network.
Reprint: https://www.jianshu.com/p/2b9239eb793b