iOS-VideoToolbox hardcoded H264

Foreword

VideoToolBox after the iOS8, Apple developed for hardware decoding coded H264 / H265 (support after iOS11) of the API.

For H264 do not know the children's shoes must first look at the introduction side of the H264.

Coding process

We implement a simple Demo, obtain video data from the camera, and then re-encoded to data stored in the bare H264 sandbox.

1. Create initialization VideoToolBox

 

 

 

 

The core code is as follows

- (void)initVideoToolBox {
    dispatch_sync(encodeQueue  , ^{
        frameNO = 0;
        int width = 480, height = 640; OSStatus status = VTCompressionSessionCreate(NULL, width, height, kCMVideoCodecType_H264, NULL, NULL, NULL, didCompressH264, (__bridge void *)(self), &encodingSession); NSLog(@"H264: VTCompressionSessionCreate %d", (int)status); if (status != 0) { NSLog(@"H264: Unable to create a H264 session"); return ; } // 设置实时编码输出（避免延迟） VTSessionSetProperty(encodingSession, kVTCompressionPropertyKey_RealTime, kCFBooleanTrue); VTSessionSetProperty(encodingSession, kVTCompressionPropertyKey_ProfileLevel, kVTProfileLevel_H264_Baseline_AutoLevel); // 设置关键帧（GOPsize)间隔 int frameInterval = 24; CFNumberRef frameIntervalRef = CFNumberCreate(kCFAllocatorDefault, kCFNumberIntType, &frameInterval); VTSessionSetProperty(encodingSession, kVTCompressionPropertyKey_MaxKeyFrameInterval, frameIntervalRef); //设置期望帧率 int fps = 24; CFNumberRef fpsRef = CFNumberCreate(kCFAllocatorDefault, kCFNumberIntType, &fps); VTSessionSetProperty(encodingSession, kVTCompressionPropertyKey_ExpectedFrameRate, fpsRef); //设置码率，均值，单位是byte int bitRate = width * height * 3 * 4 * 8; CFNumberRef bitRateRef = CFNumberCreate(kCFAllocatorDefault, kCFNumberSInt32Type, &bitRate); VTSessionSetProperty(encodingSession, kVTCompressionPropertyKey_AverageBitRate, bitRateRef); //设置码率，上限，单位是bps int bitRateLimit = width * height * 3 * 4; CFNumberRef bitRateLimitRef = CFNumberCreate(kCFAllocatorDefault, kCFNumberSInt32Type, &bitRateLimit); VTSessionSetProperty(encodingSession, kVTCompressionPropertyKey_DataRateLimits, bitRateLimitRef); //开始编码 VTCompressionSessionPrepareToEncodeFrames(encodingSession); }); } 

Initialization coding type set here kCMVideoCodecType_H264,
a resolution of 640 * 480, fps, GOP, code rate.

2. Get the data from the video camera is encoded into H264 threw VideoToolBox

 

 

 

 

 

Initializing core code is as follows video capture terminal

//初始化摄像头采集端
- (void)initCapture{
    
    self.captureSession = [[AVCaptureSession alloc]init];
    
    //设置录制640 * 480 self.captureSession.sessionPreset = AVCaptureSessionPreset640x480; AVCaptureDevice *inputCamera = [self cameraWithPostion:AVCaptureDevicePositionBack]; self.captureDeviceInput = [[AVCaptureDeviceInput alloc] initWithDevice:inputCamera error:nil]; if ([self.captureSession canAddInput:self.captureDeviceInput]) { [self.captureSession addInput:self.captureDeviceInput]; } self.captureDeviceOutput = [[AVCaptureVideoDataOutput alloc] init]; [self.captureDeviceOutput setAlwaysDiscardsLateVideoFrames:NO]; //设置YUV420p输出 [self.captureDeviceOutput setVideoSettings:[NSDictionary dictionaryWithObject:[NSNumber numberWithInt:kCVPixelFormatType_420YpCbCr8BiPlanarFullRange] forKey:(id)kCVPixelBufferPixelFormatTypeKey]]; [self.captureDeviceOutput setSampleBufferDelegate:self queue:captureQueue]; if ([self.captureSession canAddOutput:self.captureDeviceOutput]) { [self.captureSession addOutput:self.captureDeviceOutput]; } //建立连接 AVCaptureConnection *connection = [self.captureDeviceOutput connectionWithMediaType:AVMediaTypeVideo]; [connection setVideoOrientation:AVCaptureVideoOrientationPortrait]; } 

It should be noted consistent provided video resolution, and an encoder 640 * 480. AVCaptureVideoDataOutput type selection YUV420p.

Camera data portion callback

- (void)captureOutput:(AVCaptureOutput *)output didOutputSampleBuffer:(CMSampleBufferRef)sampleBuffer fromConnection:(AVCaptureConnection *)connection{
    dispatch_sync(encodeQueue, ^{ [self encode:sampleBuffer]; }); } //编码sampleBuffer - (void) encode:(CMSampleBufferRef )sampleBuffer { CVImageBufferRef imageBuffer = (CVImageBufferRef)CMSampleBufferGetImageBuffer(sampleBuffer); // 帧时间，如果不设置会导致时间轴过长。 CMTime presentationTimeStamp = CMTimeMake(frameNO++, 1000); VTEncodeInfoFlags flags; OSStatus statusCode = VTCompressionSessionEncodeFrame(encodingSession, imageBuffer, presentationTimeStamp, kCMTimeInvalid, NULL, NULL, &flags); if (statusCode != noErr) { NSLog(@"H264: VTCompressionSessionEncodeFrame failed with %d", (int)statusCode); VTCompressionSessionInvalidate(encodingSession); CFRelease(encodingSession); encodingSession = NULL; return; } NSLog(@"H264: VTCompressionSessionEncodeFrame Success"); } 

3. The frame data structure appearing
CMSampleBufferRef
storing one or more compressed or uncompressed media data;
the FIG include two CMSampleBuffer.

 

 

 

CMTime
64-bit value, 32-bit scale, media time format;

 

 

 

 

 

CMBlockBuffer
here called bare data;

CVPixelBuffer
comprising uncompressed pixel data, the image width, height;

pixelBufferAttributes
CFDictionary including width and height, pixel format (RGBA, YUV), use scenarios (OpenGL ES, Core Animation)

 

 

 

CVPixelBufferPool
CVPixelBuffer buffer pool because of the high cost of creation and destruction of CVPixelBuffer

 

 

 

 

CMVideoFormatDescription
Video formats, including width and height, color space, encoding format information and the like; for H264, and further comprising pps sps transactions;

 

 

 

 

4. The encoded data after the completion of the writing H264

 

 

 

 

Here we first complete coding judge whether the I-frame, and if it is necessary to read sps pps parameter sets, why should this happen?

We look at a bare data H264 (Elementary Stream) composed of NALU

 

 

 

 

 

Bare H.264 stream, a separate SPS, PPS absence of packets or frames, but additional I frame front, generally in the form of stored

00 00 00 01 SPS 00 00 00 01 PPS 00 00 00 01 I帧

00 00 The foregoing data is called a start code (Start Code), they do not belong SPS, the PPS content.

SPS (Sequence Parameter Sets) and PPS (Picture Parameter Set): H.264 of the SPS and PPS includes information parameter initialization required H.264 decoder, including width and height in coding the profile, level, image, deblocking character filters.

Sps are described above and are packaged in CMFormatDescriptionRef pps. So we have to remove and pps sps stream CMFormatDescriptionRef written in bare h264.

It is not difficult to understand the flow of the H264 is written.

code show as below

// 编码完成回调
void didCompressH264(void *outputCallbackRefCon, void *sourceFrameRefCon, OSStatus status, VTEncodeInfoFlags infoFlags, CMSampleBufferRef sampleBuffer) { NSLog(@"didCompressH264 called with status %d infoFlags %d", (int)status, (int)infoFlags); if (status != 0) { return; } if (!CMSampleBufferDataIsReady(sampleBuffer)) { NSLog(@"didCompressH264 data is not ready "); return; } ViewController* encoder = (__bridge ViewController*)outputCallbackRefCon; bool keyframe = !CFDictionaryContainsKey( (CFArrayGetValueAtIndex(CMSampleBufferGetSampleAttachmentsArray(sampleBuffer, true), 0)), kCMSampleAttachmentKey_NotSync); // 判断当前帧是否为关键帧 // 获取sps & pps数据 if (keyframe) { CMFormatDescriptionRef format = CMSampleBufferGetFormatDescription(sampleBuffer); size_t sparameterSetSize, sparameterSetCount; const uint8_t *sparameterSet; OSStatus statusCode = CMVideoFormatDescriptionGetH264ParameterSetAtIndex(format, 0, &sparameterSet, &sparameterSetSize, &sparameterSetCount, 0 ); if (statusCode == noErr) { // 获得了sps，再获取pps size_t pparameterSetSize, pparameterSetCount; const uint8_t *pparameterSet; OSStatus statusCode = CMVideoFormatDescriptionGetH264ParameterSetAtIndex(format, 1, &pparameterSet, &pparameterSetSize, &pparameterSetCount, 0 ); if (statusCode == noErr) { // 获取SPS和PPS data NSData *sps = [NSData dataWithBytes:sparameterSet length:sparameterSetSize]; NSData *pps = [NSData dataWithBytes:pparameterSet length:pparameterSetSize]; if (encoder) { [encoder gotSpsPps:sps pps:pps]; } } } } CMBlockBufferRef dataBuffer = CMSampleBufferGetDataBuffer(sampleBuffer); size_t length, totalLength; char *dataPointer; //这里获取了数据指针，和NALU的帧总长度，前四个字节里面保存的 OSStatus statusCodeRet = CMBlockBufferGetDataPointer(dataBuffer, 0, &length, &totalLength, &dataPointer); if (statusCodeRet == noErr) { size_t bufferOffset = 0; static const int AVCCHeaderLength = 4; // 返回的nalu数据前四个字节不是0001的startcode，而是大端模式的帧长度length // 循环获取nalu数据 while (bufferOffset < totalLength - AVCCHeaderLength) { uint32_t NALUnitLength = 0; // 读取NALU长度的数据 memcpy(&NALUnitLength, dataPointer + bufferOffset, AVCCHeaderLength); // 从大端转系统端 NALUnitLength = CFSwapInt32BigToHost(NALUnitLength); NSData* data = [[NSData alloc] initWithBytes:(dataPointer + bufferOffset + AVCCHeaderLength) length:NALUnitLength]; [encoder gotEncodedData:data]; // 移动到下一个NALU单元 bufferOffset += AVCCHeaderLength + NALUnitLength; } } } //填充SPS和PPS数据 - (void)gotSpsPps:(NSData*)sps pps:(NSData*)pps { NSLog(@"gotSpsPps %d %d", (int)[sps length], (int)[pps length]); const char bytes[] = "\x00\x00\x00\x01"; size_t length = (sizeof bytes) - 1; //string literals have implicit trailing '\0' NSData *ByteHeader = [NSData dataWithBytes:bytes length:length]; //写入startcode [self.h264FileHandle writeData:ByteHeader]; [self.h264FileHandle writeData:sps]; //写入startcode [self.h264FileHandle writeData:ByteHeader]; [self.h264FileHandle writeData:pps]; } //填充NALU数据 - (void)gotEncodedData:(NSData*)data { NSLog(@"gotEncodedData %d", (int)[data length]); if (self.h264FileHandle != NULL) { const char bytes[] = "\x00\x00\x00\x01"; size_t length = (sizeof bytes) - 1; //string literals have implicit trailing '\0' NSData *ByteHeader = [NSData dataWithBytes:bytes length:length]; //写入startcode [self.h264FileHandle writeData:ByteHeader]; //写入NALU数据 [self.h264FileHandle writeData:data]; } } 

After the destruction of coding session

- (void)EndVideoToolBox
{
    VTCompressionSessionCompleteFrames(encodingSession, kCMTimeInvalid);
    VTCompressionSessionInvalidate(encodingSession);
    CFRelease(encodingSession);
    encodingSession = NULL;
}

This completes the use VideoToolbox of H264 encoding. Good encoding H264 file can be removed from the sandbox.

to sum up

Just do not look at the code to see the process is certainly not learn the framework, coding Try it yourself!
Demo Download: iOS-VideoToolBox-Demo