iOS Live - Streaming

Plug flow, that is, the collected audio and video data transmitted to the streaming media server through the streaming protocol.

Before extrusion flow: collection, processing, compression coding
Push the stream to do the job: the package, upload

Before you plug flow

Plug flow - the collected audio, video data transmitted to the streaming media server through the streaming protocol

Having said that, in fact, there is a library LFLiveKit has achieved the background recording, beauty features, support h264, AAC hard-coded, dynamic rate of change, RTMP transport, etc., when really we developed is very convenient to use it directly.
There is also a:

LiveVideoCoreSDK : to achieve a beauty live and filter functions, we just fill in the RTMP service address, you can directly plug flow friends.
PLCameraStreamingKit : also a good push RTMP live streaming SDK.

But still recommended LFLiveKit , and in order to learn more about this plug-flow process, first by trying to walk their own pace, to understand the next.

First, capture video

Acquisition hardware (camera) video

#import "MovieViewController.h"
#import <AVFoundation/AVFoundation.h>

@interface MovieViewController ()<AVCaptureVideoDataOutputSampleBufferDelegate,AVCaptureAudioDataOutputSampleBufferDelegate>

@property (nonatomic, strong) AVCaptureSession *session;
@property (nonatomic, strong) AVCaptureVideoDataOutput *videoOutput;
@property (nonatomic, strong) AVCaptureAudioDataOutput *audioOutput;

@property (nonatomic, strong) dispatch_queue_t videoQueue;
@property (nonatomic, strong) dispatch_queue_t audioQueue;

@property (nonatomic, strong) AVCaptureConnection *videoConnection;
@property (nonatomic, strong) AVCaptureConnection *audioConnection;

@property (nonatomic, strong) AVCaptureVideoPreviewLayer *previewLayer;

@end

@implementation MovieViewController

- (void)viewDidLoad {
    [super viewDidLoad];
    [self initSession];
    [self showPlayer];
}

- (void)viewWillAppear:(BOOL)animated {
    [super viewWillAppear:animated];
    [self.session startRunning];
}

- (void)viewDidDisappear:(BOOL)animated {
    [self.session stopRunning];
}

- (void)initSession {
    // 初始化 session
    _session = [[AVCaptureSession alloc] init];

    // 配置采集输入源（摄像头）
    NSError *error = nil;
    // 获得一个采集设备, 默认后置摄像头
    AVCaptureDevice *videoDevice = [AVCaptureDevice defaultDeviceWithMediaType:AVMediaTypeVideo];
    AVCaptureDevice *audioDevice = [AVCaptureDevice defaultDeviceWithMediaType:AVMediaTypeAudio];
    // 用设备初始化一个采集的输入对象
    AVCaptureDeviceInput *videoInput = [AVCaptureDeviceInput deviceInputWithDevice:videoDevice error:&error];
    AVCaptureDeviceInput *audioInput = [AVCaptureDeviceInput deviceInputWithDevice:audioDevice error:&error];
    if (error) {
        NSLog(@"Error getting  input device: %@", error.description);
        return;
    }

    if ([_session canAddInput:videoInput]) {
        [_session addInput:videoInput]; // 添加到Session
    }
    if ([_session canAddInput:audioInput]) {
        [_session addInput:audioInput]; // 添加到Session
    }
    // 配置采集输出，即我们取得视频图像的接口
    _videoQueue = dispatch_queue_create("Video Capture Queue", DISPATCH_QUEUE_SERIAL);
    _audioQueue = dispatch_queue_create("Audio Capture Queue", DISPATCH_QUEUE_SERIAL);

    _videoOutput = [[AVCaptureVideoDataOutput alloc] init];
    _audioOutput = [[AVCaptureAudioDataOutput alloc] init];

    [_videoOutput setSampleBufferDelegate:self queue:_videoQueue];
    [_audioOutput setSampleBufferDelegate:self queue:_audioQueue];

    // 配置输出视频图像格式
    NSDictionary *captureSettings = @{(NSString*)kCVPixelBufferPixelFormatTypeKey: @(kCVPixelFormatType_32BGRA)};
    _videoOutput.videoSettings = captureSettings;
    _videoOutput.alwaysDiscardsLateVideoFrames = YES;
    if ([_session canAddOutput:_videoOutput]) {
       [_session addOutput:_videoOutput];  // 添加到Session
    }

    if ([_session canAddOutput:_audioOutput]) {
        [_session addOutput:_audioOutput]; // 添加到Session
    }
    // 保存Connection，用于在SampleBufferDelegate中判断数据来源（Video/Audio）
    _videoConnection = [_videoOutput connectionWithMediaType:AVMediaTypeVideo];
    _audioConnection = [_audioOutput connectionWithMediaType:AVMediaTypeAudio];

}

- (void)showPlayer {
    _previewLayer = [AVCaptureVideoPreviewLayer layerWithSession:_session];
    _previewLayer.videoGravity = AVLayerVideoGravityResizeAspectFill; // 设置预览时的视频缩放方式
    [[_previewLayer connection] setVideoOrientation:AVCaptureVideoOrientationPortrait]; // 设置视频的朝向
    _previewLayer.frame = self.view.layer.bounds;
    [self.view.layer addSublayer:_previewLayer];
}

#pragma mark 获取 AVCapture Delegate
- (void)captureOutput:(AVCaptureOutput *)captureOutput didOutputSampleBuffer:(CMSampleBufferRef)sampleBuffer fromConnection:(AVCaptureConnection *)connection {

    // 这里的sampleBuffer就是采集到的数据了,根据connection来判断，是Video还是Audio的数据
    if (connection == self.videoConnection) {  // Video
        NSLog(@"这里获的 video sampleBuffer，做进一步处理（编码H.264）");
    } else if (connection == self.audioConnection) {  // Audio
        NSLog(@"这里获得 audio sampleBuffer，做进一步处理（编码AAC）");
    }
}

@end

The above is roughly the achievement of the basic data acquisition, some of the details (size, direction) being not deeply, truly do live time, usually video and audio are treated separately , only important to note that the proxy method.

Two, GPUImage processing

Before coding H.264, in general we will certainly do some beauty treatment, otherwise it aired feel too real, a little ugly friends, in this example a brief whitening and dermabrasion to understand. (Specific reference is: Kun Jun -based GPUImage beauty of the real-time filters )

Directly BeautifyFaceDemo in the class GPUImageBeautifyFilter, the picture can be processed directly:

GPUImageBeautifyFilter *filter = [[GPUImageBeautifyFilter alloc] init];
UIImage *image = [UIImage imageNamed:@"testMan"];
UIImage *resultImage = [filter imageByFilteringImage:image];
self.backgroundView.image = resultImage;

NOTES CMSampleBufferRef with UIImage conversion

- (UIImage *)sampleBufferToImage:(CMSampleBufferRef)sampleBuffer {
    //制作 CVImageBufferRef
    CVImageBufferRef buffer;
    buffer = CMSampleBufferGetImageBuffer(sampleBuffer);

    CVPixelBufferLockBaseAddress(buffer, 0);

    //从 CVImageBufferRef 取得影像的细部信息
    uint8_t *base;
    size_t width, height, bytesPerRow;
    base = CVPixelBufferGetBaseAddress(buffer);
    width = CVPixelBufferGetWidth(buffer);
    height = CVPixelBufferGetHeight(buffer);
    bytesPerRow = CVPixelBufferGetBytesPerRow(buffer);

    //利用取得影像细部信息格式化 CGContextRef
    CGColorSpaceRef colorSpace;
    CGContextRef cgContext;
    colorSpace = CGColorSpaceCreateDeviceRGB();
    cgContext = CGBitmapContextCreate(base, width, height, 8, bytesPerRow, colorSpace, kCGBitmapByteOrder32Little | kCGImageAlphaPremultipliedFirst);
    CGColorSpaceRelease(colorSpace);

    //透过 CGImageRef 将 CGContextRef 转换成 UIImage
    CGImageRef cgImage;
    UIImage *image;
    cgImage = CGBitmapContextCreateImage(cgContext);
    image = [UIImage imageWithCGImage:cgImage];
    CGImageRelease(cgImage);
    CGContextRelease(cgContext);
    CVPixelBufferUnlockBaseAddress(buffer, 0);
    return image;
}

But the video is how to perform cosmetic deal with it? How to convert it? Normally we use this directly:

GPUImageBeautifyFilter *beautifyFilter = [[GPUImageBeautifyFilter alloc] init];
[self.videoCamera addTarget:beautifyFilter];
[beautifyFilter addTarget:self.gpuImageView];

Here uses GPUImageVideoCamera , an overview of the next GPUImage detailed analysis of the (three) - Real-time beauty Filters :

GPUImageVideoCamera : GPUImageOutput subclass providing image data from the camera as the data source, usually a source of responder chain.
GPUImageView : in response to the end of the chain, is generally used for displaying the image GPUImage.
GPUImageFilter : for receiving the source image, custom vertex, fragment shader to render a new image, and object response notice a strand after drawing.
GPUImageFilterGroup : a collection of more than GPUImageFilter.

GPUImageBeautifyFilter：

@interface GPUImageBeautifyFilter : GPUImageFilterGroup {
  GPUImageBilateralFilter *bilateralFilter;
  GPUImageCannyEdgeDetectionFilter *cannyEdgeFilter;
  GPUImageCombinationFilter *combinationFilter;
  GPUImageHSBFilter *hsbFilter;
}

Simple to understand the beauty of the process

GPUImage have to say is quite powerful, function here only show a small portion, wherein the processing of personal piece of filter there are still many do not understand, you need to learn more about eating source, but more temporarily introduced. After this process sampleBuffer by cosmetic treatment, it is naturally encoded.

Third, video, audio compression coding

The coding is hard-coded it or soft-coded it? The same rate, the image quality soft knitting sharper, but higher power consumption, and can lead to over-blanching CPU to the camera. But hard-coded involve decoding other platforms, there are many pit. In summary, iOS end hardware compatibility is better, iOS 8.0 share is already high, it can be directly used hardcoded.

Hard coded: DEMO the next few contrast can, of course, see LFLiveKit more directly.

Soft encoding: using the FFmpeg + x264 iOS camera live video stream is encoded into h264 file , the notes: the FFmpeg--for-the Encode the X264-iOS

I used directly LFLiveKit , which has been packaged very good friends, here on Audiotoolbox && VideoToolbox simple to understand the next:

AudioToolbox
iOS use AudioToolbox AudioConverter API converts the source format to a target format, can be seen in detail using the AAC encoder iOS own .

// 1、根据输入样本初始化一个编码转换器
AudioStreamBasicDescription 根据指定的源格式和目标格式创建 audio converter
// 2、初始化一个输出缓冲列表 outBufferList 
// 3、获取 AudioCallBack
OSStatus inputDataProc(AudioConverterRef inConverter, UInt32 *ioNumberDataPackets, AudioBufferList *ioData,AudioStreamPacketDescription **outDataPacketDescription, void *inUserData) 
// 4、音频格式完成转换
AudioConverterFillComplexBuffer 实现inBufferList 和 outBufferList、inputDataProc音频格式之间的转换。

VideoToolbox
hard decoding after iOS8, hard coded API, where only encoding.

// 1、初始化 VTCompressionSessionRef  
- (void)initCompressionSession;
// 2、传入  解码一个frame
VTCompressionSessionEncodeFrame(compressionSession, pixelBuffer, presentationTimeStamp, duration, (__bridge CFDictionaryRef)properties, (__bridge_retained void *)timeNumber, &flags);
// 3、回调，处理 取得PPS和SPS
static void VideoCompressonOutputCallback(void *VTref, void *VTFrameRef, OSStatus status, VTEncodeInfoFlags infoFlags, CMSampleBufferRef sampleBuffer)
// 4、完成编码，然后销毁session
VTCompressionSessionCompleteFrames(compressionSession, kCMTimeInvalid);
VTCompressionSessionInvalidate(compressionSession);
CFRelease(compressionSession);
compressionSession = NULL;

Fourth, plug flow

Encapsulating the data into FLV, upload package through RTMP protocol, i.e. the server from the anchor end plug flow is substantially complete.

4-1, the package data is typically encapsulated in FLV

FLV streaming media format is a new video format, called FlashVideo. Because it forms a very small file, loads fast, making the network become possible to watch video files, it appears to effectively address the video files into Flash after the exported SWF file bulky, not a good use on the network and other shortcomings. (What)

Format: from ( Packet FLV )

一般FLV 文件结构里是这样存放的：
[[Flv Header]
[Metainfo Tag]
[Video Tag]
[Audio Tag]
[Video Tag]
[Audio Tag]
[Other Tag]…]
其中 AudioTag 和 VideoTag 出现的顺序随机的，没有严格的定义。
Flv Header 是文件的头部，用FLV字符串标明了文件的类型，以及是否有音频、视频等信息。之后会有几个字节告诉接下来的包字节数。
Metainfo 中用来描述Flv中的各种参数信息，例如视频的编码格式、分辨率、采样率等等。如果是本地文件（非实时直播流），还会有偏移时间戳之类的信息用于支持快进等操作。
VideoTag 存放视频数据。对于H.264来说，第一帧发送的NALU应为 SPS和PPS，这个相当于H.264的文件头部，播放器解码流必须先要找到这个才能进行播放。之后的数据为I帧或P帧。
AudioTag 存放音频数据。对于AAC来说，我们只需要在每次硬编码完成后给数据加上adts头部信息即可。

IOS use in: a detailed look at LFStreamRTMPSocket class LFLiveKit in.

4-2、RTMP

Plug flow from end to end service, data is processed, the most commonly used protocol is RTMP (Real Time Messaging Protocol, Real Time Messaging Protocol).

RTMP的传输延迟通常在1-3秒，符合手机直播对性能的要求，因此RTMP是手机直播中最常见的传输协议。

但是网络延迟和阻塞等问题的一直存在的，所以通过Quality of Servic一种网络机制将流数据推送到网络端，通过CDN分发是必要的。

另外，服务端还需要对数据流一定的处理，转码，使得数据流支持HLS，HTTP-FLV，RTMP等格式的拉流，支持一转多，适配不同网络、分辨率的终端。（当然这就是服务端要做的事情啦）

Can LFLiveKit directly try, or you can take a look at LMLiveStreaming , of course, here first with a local video push it a try.

4-3, plug flow local analog

Here is followed by the rapid integration of iOS-based video streaming RTMP of push to achieve, or even the basic display can not be friends ah. Here it can also be combined with the Mac build nginx + rtmp server to install, after installing nginx, install ffmpeg, download VLC you can directly start it

At first when using ffmpeg I encountered the following error:

A wrong input

Later he found himself entered wrong, or carefully:
video file address: /Users/qiu/Desktop/kobe.mp4 (own a test video)
push pull stream flow address: rtmp: // localhost: 1935 / rtmplive / room

~ ffmpeg -re -i /Users/qiu/Desktop/kobe -vcodec libx264 -acodec aac -strict -2 -f flv rtmp://localhost:1935/rtmplive/room

That -vcodec libx264 -acodec aac -strict -2 -f flvcommand is also not wrong, ffmpeg command reference FFmpeg commonly used basic commands .

kobeAndOneal.gif

4-4, the phone live - display on VLC

In order to feel better at, we can directly use LMLiveStreaming , and then open in VLC file - Open Network, enter the code directly in the url:

The code of this address

Then we end computer can display it

Live.gif

At present, there are 2 seconds delay, saying it is normal. But how to optimize it? I do not know if there are good suggestions welcome friends advertised. Under Note: optimization of broadcast delay accumulated .

to sum up

PS: The above transport stream just push server end of the simulation, however, the transmission system generally comprises a plurality of portions connected end plug flow, a plurality of service end portions, the end play and the like. The iOS players end piece directly ijkplayer , like on a note - On the broadcast , quickly realized the process of pulling the stream, of course, ijkplayer too strong reason slightly.

The following macro level understanding of the entire transmission process at:

The overall transmission process

PS: Also in fact a lot of third-party integration as well, refer to

Overall, this is a rough process, a good number of standing on the shoulders of giants, but still a basic understanding of the process flow of a push, no formal experience in the project, there must be too many minutiae ignored and a lot of pit need to fill, or that purpose, as being the first point of their prior knowledge of it, but here can be extended and in-depth knowledge really matter too much, as LFLiveKit and GPUImage only expose the tip of the iceberg.

Note Reference:

iOS Live - Streaming

Remapped plug flow off LFLiveKit

[How to quickly develop a complete iOS app Live] (principle chapter)

FFmpeg simple push-based flow (Case in a push RTMP)

Based GPUImage beauty of the real-time filters