iOS development - WebRTC local live high resolution does not display the screen problem
When using WebRTC combined with ossrs to push the stream before, the player of ossrs could not see the problem of high-resolution images. According to this question, a solution was found.
1. What is WebRTC
What is WebRTC?
WebRTC (Web Real-Time Communications) is a real-time communication technology that allows network applications or sites to establish a peer-to-peer (Peer-to-Peer) connection between browsers without intermediaries to achieve video streaming and/or the transmission of audio streams or other arbitrary data.
2. What is ossrs?
what is ossrs
SRS (Simple Realtime Server) is a simple and efficient real-time video server that supports multiple real-time streaming media protocols such as RTMP, WebRTC, HLS, HTTP-FLV, and SRT.
Official website address: https://ossrs.net/lts/zh-cn/
Here I will not write about iOS Google WebRTC and ossrs to achieve RTC live broadcast. For the time being, it is worth recording the problem that the high-resolution local WebRTC live broadcast does not display the picture.
3. Solution to the problem of no display screen in high resolution
The problem that the local WebRTC live high-resolution live broadcast does not display the picture is related to the profile-level-id in the SDP.
profile-level-id is exactly the base16 encoding of the second to four bytes in SPS. The specific meaning of these three bytes is
sps[1] AVCProfileIndication
sps[2] profile_compatibility
sps[3] AVCLevlIndication
When http://en.wikipedia.org/wiki/H.264/MPEG-4_AVC#Levels
is actually set, the level value is multiplied by 10, for example, level 1.0, the set value is 0x0A. level 3.0, the setting value is 0x1E. The exception is level 1b, the setting value is 0x09
Image source network (sorry forgot the address, if you quote your blog image, please leave a message)
The third represents level. For example, the value of 1f is 31. It can be seen from the figure that it is 3.1, and the resolution is 720 480 720 576 1280*720
If you need high resolution 1920 1080 2560 1920 3840*2160, you need to set the level 5.1 hexadecimal value to 33, the value set here is 42e033.
Below is an example SDP
“code”: 0,
“server”: “vid-415v5lz”,
“sdp”: “v=0\r\no=SRS/4.0.268(Leo) 94003279212192 2 IN IP4 0.0.0.0\r\ns=SRSPublishSession\r\nt=0 0\r\na=ice-lite\r\na=group:BUNDLE 0 1\r\na=msid-semantic: WMS live/livestream\r\nm=audio 9 UDP/TLS/RTP/SAVPF 111\r\nc=IN IP4 0.0.0.0\r\na=ice-ufrag:q01184s8\r\na=ice-pwd:o25158210twbb093o342910094v0wo5k\r\na=fingerprint:sha-256 6A:66:81:7C:68:91:79:18:05:2C:EE:5F:BF:1B:4B:F4:78:C4:01:06:CC:CC:9E:F0:32:5B:72:21:4A:C2:A1:AA\r\na=setup:passive\r\na=mid:0\r\na=recvonly\r\na=rtcp-mux\r\na=rtcp-rsize\r\na=rtpmap:111 opus/48000/2\r\na=fmtp:111 minptime=10;useinbandfec=1\r\na=candidate:0 1 udp 2130706431 10.0.80.128 8000 typ host generation 0\r\na=candidate:1 1 udp 2130706431 112.124.157.141 8000 typ host generation 0\r\nm=video 9 UDP/TLS/RTP/SAVPF 96 127\r\nc=IN IP4 0.0.0.0\r\na=ice-ufrag:q01184s8\r\na=ice-pwd:o25158210twbb093o342910094v0wo5k\r\na=fingerprint:sha-256 6A:66:81:7C:68:91:79:18:05:2C:EE:5F:BF:1B:4B:F4:78:C4:01:06:CC:CC:9E:F0:32:5B:72:21:4A:C2:A1:AA\r\na=setup:passive\r\na=mid:1\r\na=extmap:5 http://www.ietf.org/id/draft-holmer-rmcat-transport-wide-cc-extensions-01\r\na=recvonly\r\na=rtcp-mux\r\na=rtcp-rsize\r\na=rtpmap:96 H264/90000\r\na=rtcp-fb:96 transport-cc\r\na=rtcp-fb:96 nack\r\na=rtcp-fb:96 nack pli\r\na=fmtp:96 level-asymmetry-allowed=1;packetization-mode=1;profile-level-id=42e01f\r\na=rtpmap:127 red/90000\r\na=candidate:0 1 udp 2130706431 10.0.80.128 8000 typ host generation 0\r\na=candidate:1 1 udp 2130706431 112.124.157.141 8000 typ host generation 0\r\n”,
“sessionid”: “q01184s8:oPvh”
}
You can see profile-level-id=42e01f here
If you replace the profile-level-id of the local SDP with 42e033 in WebRTC
[weakSelf.webRTCClient offer:^(RTCSessionDescription *sdp) {
DebugLog(@"changeSDP2Server offer sdp:%@", sdp);
NSString *offerSDPString = [SDWebRTCSDPUtil setMediaBitrate:sdp.sdp media:@"video" bitrate:(6*1024*1024)];
DebugLog(@"changeSDP2Server offerSDPString:%@", offerSDPString);
[weakSelf changeSDP2Server:offerSDPString];
}];
Change the profile-level-id in the remoteSDPString obtained by calling the rtc/v1/publish/ interface to 42e033 before calling
- (void)setRemoteDescription:(RTC_OBJC_TYPE(RTCSessionDescription) *)sdp
completionHandler:(nullable void (^)(NSError *_Nullable error))completionHandler;
After testing, switching high-resolution images can be displayed normally.
NSString *resultRemoteSDPString = [SDWebRTCSDPUtil setMediaBitrate:remoteSDPString media:@"video" bitrate:(6*1024*1024)];
DebugLog(@"changeSDP2Server resultRemoteSDPString:%@", resultRemoteSDPString);
RTCSessionDescription *remoteSDP = [[RTCSessionDescription alloc] initWithType:RTCSdpTypeAnswer sdp:resultRemoteSDPString];
[weakSelf.webRTCClient setRemoteSdp:remoteSDP completion:^(NSError * error) {
DebugLog(@"changeSDP2Server setRemoteDescription error:%@", error);
}];
4. Solve the problem of high resolution not displaying the screen through RTCVideoEncoderFactory
Changing the profile-level-id in SDP is not my final solution. I finally use the profile-level-id value of constrainedHighParams in RTCVideoCodecInfo of RTCVideoEncoder to specify 42e033
Set the constrainedHighInfo and constrainedBaselineInfo elements of the codecs.
NSDictionary<NSString *, NSString *> *constrainedHighParams = @{
@"profile-level-id" : kLevelHighConstrainedHigh,
@"level-asymmetry-allowed" : @"1",
@"packetization-mode" : @"1",
};
RTCVideoCodecInfo *constrainedHighInfo =
[[RTCVideoCodecInfo alloc] initWithName:kRTCVideoCodecH264Name parameters:constrainedHighParams];
[codecs addObject:constrainedHighInfo];
NSDictionary<NSString *, NSString *> *constrainedBaselineParams = @{
@"profile-level-id" : kLevelHighConstrainedBaseline,
@"level-asymmetry-allowed" : @"1",
@"packetization-mode" : @"1",
};
RTCVideoCodecInfo *constrainedBaselineInfo =
[[RTCVideoCodecInfo alloc] initWithName:kRTCVideoCodecH264Name parameters:constrainedBaselineParams];
[codecs addObject:constrainedBaselineInfo];
The complete code is as follows
SDRTCVideoEncoderFactory.h
#import <Foundation/Foundation.h>
#import <WebRTC/WebRTC.h>
/**
+ (NSArray<RTCVideoCodecInfo *> *)supportedCodecs {
NSDictionary<NSString *, NSString *> *constrainedHighParams = @{
@"profile-level-id" : kRTCMaxSupportedH264ProfileLevelConstrainedHigh,
@"level-asymmetry-allowed" : @"1",
@"packetization-mode" : @"1",
};
RTCVideoCodecInfo *constrainedHighInfo =
[[RTCVideoCodecInfo alloc] initWithName:kRTCVideoCodecH264Name
parameters:constrainedHighParams];
NSDictionary<NSString *, NSString *> *constrainedBaselineParams = @{
@"profile-level-id" : kRTCMaxSupportedH264ProfileLevelConstrainedBaseline,
@"level-asymmetry-allowed" : @"1",
@"packetization-mode" : @"1",
};
RTCVideoCodecInfo *constrainedBaselineInfo =
[[RTCVideoCodecInfo alloc] initWithName:kRTCVideoCodecH264Name
parameters:constrainedBaselineParams];
RTCVideoCodecInfo *vp8Info = [[RTCVideoCodecInfo alloc] initWithName:kRTCVideoCodecVp8Name];
#if defined(RTC_ENABLE_VP9)
RTCVideoCodecInfo *vp9Info = [[RTCVideoCodecInfo alloc] initWithName:kRTCVideoCodecVp9Name];
#endif
return @[
constrainedHighInfo,
constrainedBaselineInfo,
vp8Info,
#if defined(RTC_ENABLE_VP9)
vp9Info,
#endif
];
}
*/
@interface SDRTCVideoEncoderFactory : NSObject<RTCVideoEncoderFactory>
@end
SDRTCVideoEncoderFactory.m
#import "SDRTCVideoEncoderFactory.h"
static NSString *kLevelHighConstrainedHigh = @"640c33";
static NSString *kLevelHighConstrainedBaseline = @"42e033";
@implementation SDRTCVideoEncoderFactory
- (id<RTCVideoEncoder>)createEncoder:(RTCVideoCodecInfo *)info {
if ([info.name isEqualToString:kRTCVideoCodecH264Name]) {
return [[RTCVideoEncoderH264 alloc] initWithCodecInfo:info];
} else if ([info.name isEqualToString:kRTCVideoCodecVp8Name]) {
return [RTCVideoEncoderVP8 vp8Encoder];
} else if ([info.name isEqualToString:kRTCVideoCodecVp9Name]) {
return [RTCVideoEncoderVP9 vp9Encoder];
}
return nil;
}
- (NSArray<RTCVideoCodecInfo *> *)supportedCodecs {
NSMutableArray<RTCVideoCodecInfo *> *codecs = [NSMutableArray array];
NSDictionary<NSString *, NSString *> *constrainedHighParams = @{
@"profile-level-id" : kLevelHighConstrainedHigh,
@"level-asymmetry-allowed" : @"1",
@"packetization-mode" : @"1",
};
RTCVideoCodecInfo *constrainedHighInfo =
[[RTCVideoCodecInfo alloc] initWithName:kRTCVideoCodecH264Name parameters:constrainedHighParams];
[codecs addObject:constrainedHighInfo];
NSDictionary<NSString *, NSString *> *constrainedBaselineParams = @{
@"profile-level-id" : kLevelHighConstrainedBaseline,
@"level-asymmetry-allowed" : @"1",
@"packetization-mode" : @"1",
};
RTCVideoCodecInfo *constrainedBaselineInfo =
[[RTCVideoCodecInfo alloc] initWithName:kRTCVideoCodecH264Name parameters:constrainedBaselineParams];
[codecs addObject:constrainedBaselineInfo];
RTCVideoCodecInfo *vp8Info = [[RTCVideoCodecInfo alloc] initWithName:kRTCVideoCodecVp8Name parameters:nil];
[codecs addObject:vp8Info];
#if defined(RTC_ENABLE_VP9)
RTCVideoCodecInfo *vp9Info = [[RTCVideoCodecInfo alloc] initWithName:kRTCVideoCodecVp9Name];
[codecs addObject:vp9Info];
#endif
return [codecs copy];
}
@end
Finally, do some settings when RTCPeerConnectionFactory is initialized
#pragma mark - Lazy
- (RTCPeerConnectionFactory *)factory {
if (!_factory) {
RTCInitializeSSL();
SDRTCVideoDecoderFactory *decoderFactory = [[SDRTCVideoDecoderFactory alloc] init];
SDRTCVideoEncoderFactory *encoderFactory = [[SDRTCVideoEncoderFactory alloc] init];
// for (RTCVideoCodecInfo *codec in encoderFactory.supportedCodecs) {
// DebugLog(@"RTCVideoCodecInfo codec.parameters:%@", codec.parameters);
// }
_factory = [[RTCPeerConnectionFactory alloc] initWithEncoderFactory:encoderFactory decoderFactory:decoderFactory];
}
return _factory;
}
After testing, the problem of not displaying the picture in high resolution has been solved
V. Summary
iOS development - WebRTC local live broadcast with high resolution does not display the problem. When using WebRTC combined with ossrs to push the stream, the ossrs player cannot see the high resolution screen problem. Finally solved the problem by setting initWithEncoderFactory of RTCPeerConnectionFactory.
https://blog.csdn.net/gloryFlow/article/details/132240952
Learning records, keep improving every day.