The first two articles recorded some concepts and processes of audio and video calls, as well as an example of audio and video calls in a LAN.
Today, an example of audio and video calls between pseudo-real networks is used to analyze the process of WebRTC audio and video calls.
In the previous article, because they are in the same route, there is no need to go through the wall, and the two clients can directly transmit multimedia streaming data. It is also very simple to use XMPP as a channel for signaling transmission.
This article will add a STUN server and a TURN server, so that the functions of the ICE framework can be brought into play, and a complete audio and video call can be realized. However, because the network environments of the two clients are different, the two clients need to be added to the same virtual network (that is, the room server), so the support of the server is required. Regarding the development of the server, we will not describe it here. .
Process Analysis
Initiator
- The first step is still the click event of the video button, which is no different from the audio and video call in the LAN:
- (void)startCommunication:(BOOL)isVideo
{
WebRTCClient *client = [WebRTCClient sharedInstance];
client.myJID = [HLIMCenter sharedInstance].xmppStream.myJID.full;
client.remoteJID = self.chatJID.full;
[client showRTCViewByRemoteName:self.chatJID.full isVideo:isVideo isCaller:YES];
}While displaying the audio and video call view, a series of operations need to be done:
- Play the dialing sound;
- When dialing, black screen is prohibited.
- Listen to the system for incoming calls. The above steps are the same as the audio and video calls in the LAN.
The second step is to create a room in the room server and join the room. In this step, server-side personnel are required to provide a room server and handle the logic of creating and joining rooms. The client, on the other hand, randomly generates a room number, then sends a request to the room server, creates the server, and adds itself to the room; the request will return the room number (that is, the one passed in), ClientId, initiator (whether is the creator), the previous signaling message, the server-side WebSocket address and other parameters. If the room number already exists, then directly join this room. Therefore, after sending the room number to the responding party, when the responding party registers, it will only join this room and will not create a new room.
The third step is to initialize the WebRTC configuration. There are also some changes in these configurations, adding STUN and TURN servers to the ICE server. First, when the iCE server array is initialized, the STUN server is added.
instance.ICEServers = [NSMutableArray arrayWithObject:[instance defaultSTUNServer]];The creation of ICEServer has a class, RTCICEServer.
- (RTCICEServer *)defaultSTUNServer {
NSURL *defaultSTUNServerURL = [NSURL URLWithString:RTCSTUNServerURL];
return [[RTCICEServer alloc] initWithURI:defaultSTUNServerURL
username:@""
password:@""];
}You can use the STUN server provided by Google stun:stun.l.google.com:19302, or you can ask the server developer to provide a STUN server.
As for the TURN server (forwarding server), although it is generally not necessary to add it, it is better to provide several seats for backup.
/**
* 关于RTC 的设置
*/
- (void)initRTCSetting
{
//添加 turn 服务器
// NSMutableURLRequest *request = [NSMutableURLRequest requestWithURL:[NSURL URLWithString:RTCTRUNServerURL]];
// [request addValue:@"Mozilla/5.0" forHTTPHeaderField:@"user-agent"];
// [request addValue:RTCRoomServerURL forHTTPHeaderField:@"origin"];
// [request setTimeoutInterval:5];
// [request setCachePolicy:NSURLRequestReloadIgnoringCacheData];
//
// NSURLSessionDataTask *turnTask = [[NSURLSession sharedSession] dataTaskWithRequest:request completionHandler:^(NSData * _Nullable data, NSURLResponse * _Nullable response, NSError * _Nullable error) {
// NSDictionary *dict = [NSJSONSerialization JSONObjectWithData:data options:NSJSONReadingMutableContainers error:NULL];
// NSLog(@"返回的服务器：%@",dict);
// NSString *username = dict[@"username"];
// NSString *password = dict[@"password"];
// NSArray *uris = dict[@"uris"];
//
// for (NSString *uri in uris) {
// RTCICEServer *server = [[RTCICEServer alloc] initWithURI:[NSURL URLWithString:uri] username:username password:password];
// [_ICEServers addObject:server];
// }
//
// }];
// [turnTask resume];
self.peerConnection = [self.peerConnectionFactory peerConnectionWithICEServers:_ICEServers constraints:self.pcConstraints delegate:self];
//设置 local media stream
RTCMediaStream *mediaStream = [self.peerConnectionFactory mediaStreamWithLabel:@"ARDAMS"];
// 添加 local video track
RTCAVFoundationVideoSource *source = [[RTCAVFoundationVideoSource alloc] initWithFactory:self.peerConnectionFactory constraints:self.videoConstraints];
RTCVideoTrack *localVideoTrack = [[RTCVideoTrack alloc] initWithFactory:self.peerConnectionFactory source:source trackId:@"AVAMSv0"];
[mediaStream addVideoTrack:localVideoTrack];
self.localVideoTrack = localVideoTrack;
// 添加 local audio track
RTCAudioTrack *localAudioTrack = [self.peerConnectionFactory audioTrackWithID:@"ARDAMSa0"];
[mediaStream addAudioTrack:localAudioTrack];
// 添加 mediaStream
[self.peerConnection addStream:mediaStream];
RTCEAGLVideoView *localVideoView = [[RTCEAGLVideoView alloc] initWithFrame:self.rtcView.ownImageView.bounds];
localVideoView.transform = CGAffineTransformMakeScale(-1, 1);
localVideoView.delegate = self;
[self.rtcView.ownImageView addSubview:localVideoView];
self.localVideoView = localVideoView;
[self.localVideoTrack addRenderer:self.localVideoView];
RTCEAGLVideoView *remoteVideoView = [[RTCEAGLVideoView alloc] initWithFrame:self.rtcView.adverseImageView.bounds];
remoteVideoView.transform = CGAffineTransformMakeScale(-1, 1);
remoteVideoView.delegate = self;
[self.rtcView.adverseImageView addSubview:remoteVideoView];
self.remoteVideoView = remoteVideoView;
}The fourth step is to create an Offer signaling.
// 创建一个offer信令
[self.peerConnection createOfferWithDelegate:self constraints:self.sdpConstraints];- Step 5: Send the room number to the responding party, and send an offer signaling to the responding party. In the callback of creating the Offer signaling completion, if the creation is successful, send the room number to the answering party, and send the sdp of the offer to the other party.
- (void)peerConnection:(RTCPeerConnection *)peerConnection
didCreateSessionDescription:(RTCSessionDescription *)sdp
error:(NSError *)error
{
if (error) {
NSLog(@"创建SessionDescription 失败");
#warning 这里创建 创建SessionDescription 失败,创建失败应该隐藏拨打界面，并给予提示。
} else {
NSLog(@"创建SessionDescription 成功");
RTCSessionDescription *sdpH264 = [self descriptionWithDescription:sdp videoFormat:@"H264"];
[self.peerConnection setLocalDescriptionWithDelegate:self sessionDescription:sdpH264];
if ([sdp.type isEqualToString:@"offer"]) {
NSDictionary *dict = @{@"roomId":self.roomId};
NSData *data = [NSJSONSerialization dataWithJSONObject:dict options:0 error:nil];
NSString *message = [[NSString alloc] initWithData:data encoding:NSUTF8StringEncoding];
[[HLIMClient shareClient] sendSignalingMessage:message toUser:self.remoteJID];
}
NSDictionary *jsonDict = @{ @"type" : sdp.type, @"sdp" : sdp.description };
NSData *jsonData = [NSJSONSerialization dataWithJSONObject:jsonDict options:0 error:nil];
NSString *jsonStr = [[NSString alloc] initWithData:jsonData encoding:NSUTF8StringEncoding];
[[HLIMClient shareClient] sendSignalingMessage:jsonStr toUser:self.remoteJID];
}
}
In the sixth step, WebRTC interacts with the STUN server and TURN server internally. (This is a hidden operation) It is mainly reflected in several callbacks of peerConnection:
The processing of the above basic callback methods is basically the same as the previous article, and there are some changes in the two underlined callback methods. -peerConnection:iceConnectionChangedAfter disconnection is detected, remove the audio and video call interface.
key code:
case RTCICEConnectionDisconnected:
{
NSLog(@"newState = RTCICEConnectionDisconnected");
dispatch_async(dispatch_get_main_queue(), ^{
[self.rtcView dismiss];
[self cleanCache];
});
}
break;However -peerConnection:gotICECandidate, because the local end will generate candidates corresponding to different protocols for all network interfaces. Each Candidate actually describes how to communicate with itself. For example, a candidate of STUN type will include the IP and port type of the local end outside the firewall. Because of the addition of STUN and TURN servers, there are more possible communication methods, and the number of callbacks will also increase.
- In the seventh step, when XMPP receives the signaling message returned by the other party, if it is not an answer signaling, store it; if it is an answer signaling, first process the answer signaling, and then process other signaling.
- (void)receiveSignalingMessage:(NSNotification *)notification
{
NSDictionary *dict = [notification object];
[self handleSignalingMessage:dict];
[self drainMessages];
}
- (void)handleSignalingMessage:(NSDictionary *)dict
{
NSString *type = dict[@"type"];
if ([type isEqualToString:@"offer"] || [type isEqualToString:@"answer"]) {
[self.messages insertObject:dict atIndex:0];
_hasReceivedSdp = YES;
} else if ([type isEqualToString:@"candidate"]) {
[self.messages addObject:dict];
} else if ([type isEqualToString:@"bye"]) {
[self processMessageDict:dict];
}
}
- (void)drainMessages
{
if (!_peerConnection || !_hasReceivedSdp) {
return;
}
for (NSDictionary *dict in self.messages) {
[self processMessageDict:dict];
}
[self.messages removeAllObjects];
}
- (void)processMessageDict:(NSDictionary *)dict
{
NSString *type = dict[@"type"];
if ([type isEqualToString:@"offer"]) {
RTCSessionDescription *remoteSdp = [[RTCSessionDescription alloc] initWithType:type sdp:dict[@"sdp"]];
[self.peerConnection setRemoteDescriptionWithDelegate:self sessionDescription:remoteSdp];
[self.peerConnection createAnswerWithDelegate:self constraints:self.sdpConstraints];
} else if ([type isEqualToString:@"answer"]) {
RTCSessionDescription *remoteSdp = [[RTCSessionDescription alloc] initWithType:type sdp:dict[@"sdp"]];
[self.peerConnection setRemoteDescriptionWithDelegate:self sessionDescription:remoteSdp];
} else if ([type isEqualToString:@"candidate"]) {
NSString *mid = [dict objectForKey:@"id"];
NSNumber *sdpLineIndex = [dict objectForKey:@"label"];
NSString *sdp = [dict objectForKey:@"sdp"];
RTCICECandidate *candidate = [[RTCICECandidate alloc] initWithMid:mid index:sdpLineIndex.intValue sdp:sdp];
[self.peerConnection addICECandidate:candidate];
} else if ([type isEqualToString:@"bye"]) {
if (self.rtcView) {
NSData *jsonData = [NSJSONSerialization dataWithJSONObject:dict options:0 error:nil];
NSString *jsonStr = [[NSString alloc] initWithData:jsonData encoding:NSUTF8StringEncoding];
if (jsonStr.length > 0) {
[[HLIMClient shareClient] sendSignalingMessage:jsonStr toUser:self.remoteJID];
}
[self.rtcView dismiss];
[self cleanCache];
}
}
}After setting the answer signaling, the two parties can send multimedia streaming data point-to-point.
Respondent
In the first step, after receiving the room number information sent by the initiator through XMPP, the answer interface is displayed, but the RTC configuration is postponed until the answer button is clicked.
The second step is to register and join the room. Because the room has been created by the initiator, it will directly join the room.
The third step is to stop the sound playback when the answer button is clicked, and then configure the RTC.
- (void)acceptAction
{
[self.audioPlayer stop];
[self initRTCSetting];
[self drainMessages];
}The fourth step is to process the signaling message.
Before processing the signaling hours, determine whether the offer signaling has been received. If the signaling message is processed after the offer signaling is received, the sdp of the offer is now set to the remote sdp of the peerConnection. Create an answer signaling at the same time, and send the answer signaling to the peer.
After the remote and local sdp have been set up at both ends, the multimedia stream data will be sent point-to-point.
Replenish
In the first article of WebRTC, it was said that signaling can be transmitted in a variety of ways. In addition to XMPP, other protocol methods can also be used to transmit signaling, such as WebSocket. But the room number is not part of the signaling message.
How to use WebSocket to transmit signaling messages?
After the room is registered and successfully joined, the server-side WebSocket address will be returned. At this time, create a WebSocket and register with the room number and clientId. In fact, the room number and clientId are packaged and sent to the server through WebSocket.
key code:
NSURL *webSocketURL = [NSURL URLWithString:dict[kARDJoinWebSocketURLKey]];
_webSocket = [[SRWebSocket alloc] initWithURL:webSocketURL];
_webSocket.delegate = self;
[_webSocket open];
[self registerForRoomId:self.roomId clientId:self.clientId];And the key code registered with room number and clientId:
NSDictionary *registerMessage = @{
@"cmd": @"register",
@"roomid" : _roomId,
@"clientid" : _clientId,
};
NSData *message = [NSJSONSerialization dataWithJSONObject:registerMessage
options:NSJSONWritingPrettyPrinted
error:nil];
NSString *messageString = [[NSString alloc] initWithData:message encoding:NSUTF8StringEncoding];
NSLog(@"Registering on WSS for rid:%@ cid:%@", _roomId, _clientId);
// Registration can fail if server rejects it. For example, if the room is full.
[_webSocket send:messageString];Key code example for sending signaling:
NSDictionary *jsonDict = @{ @"type" : sdp.type, @"sdp" : sdp.description };
NSData *jsonData = [NSJSONSerialization dataWithJSONObject:jsonDict options:0 error:nil];
NSString *jsonStr = [[NSString alloc] initWithData:jsonData encoding:NSUTF8StringEncoding];
NSDictionary *messageDict = @{@"cmd": @"send", @"msg": jsonStr};
NSData *messageJSONObject = [NSJSONSerialization dataWithJSONObject:messageDict
options:NSJSONWritingPrettyPrinted
error:nil];
NSString *messageString = [[NSString alloc] initWithData:messageJSONObject
encoding:NSUTF8StringEncoding];
[_webSocket send:messageString];The proxy method of WebSocket will call back when the socket opens successfully, fails to open, closes, and receives a message.
Here we mainly focus on receiving messages. The received messages are signaling messages, and there are many types of signaling messages. Candidate messages need to be saved, while offer, answer, and bye messages need to be processed immediately.
The following is an example of processing received signaling messages:
- (void)webSocket:(SRWebSocket *)webSocket didReceiveMessage:(id)message {
NSString *messageString = message;
NSData *messageData = [messageString dataUsingEncoding:NSUTF8StringEncoding];
id jsonObject = [NSJSONSerialization JSONObjectWithData:messageData
options:0
error:nil];
if (![jsonObject isKindOfClass:[NSDictionary class]]) {
NSLog(@"Unexpected message: %@", jsonObject);
return;
}
NSDictionary *wssMessage = jsonObject;
NSLog(@"WebSocket 接收到信息:%@",wssMessage);
NSString *errorString = wssMessage[@"error"];
if (errorString.length) {
NSLog(@"WebSocket收到错误信息");
return;
}
NSString *msg = wssMessage[@"msg"];
NSData *data = [msg dataUsingEncoding:NSUTF8StringEncoding];
NSDictionary *sinalingMsg = [NSJSONSerialization JSONObjectWithData:data options:0 error:nil];
[self handleSignalingMessage:sinalingMsg];
[self drainMessages];
}Example project address for signaling with XMPP: RemoteXMPPRTC
Example project address for signaling with WebSocket: RemoteWebRTC
The WebRTC static library used in the project has been placed in: Baidu Netdisk
This is the end of the introduction to WebRTC, Have Fun!
Original WebRTC audio and video calls under iOS (3) - audio and video calls - Nuggets
★The business card at the end of the article can receive audio and video development learning materials for free, including (FFmpeg, webRTC, rtmp, hls, rtsp, ffplay, srs) and audio and video learning roadmaps, etc.
see below!