H264 hardware solution and display instructions in iOS system
Before the iOS 8.0 system, Apple did not open the hardware encoding and decoding function of the system, but the Mac OS system has always had a framework called Video ToolBox to handle hardware encoding and decoding. Finally, after iOS 8.0, Apple introduced this framework to iOS. system.
1. Basic data structure of VideoToolbox:
1. CVPixelBuffer: image data structure before encoding and after decoding;
2. CMTime, CMClock and CMTimebase: Timestamp related. The time appears in the form of 64-bit/32-bit;
3. CMBlockBuffer: After encoding, the data structure of the resulting image;
4. CMVideoFormatDescription: image storage method, codec and other format descriptions;
5. CMSampleBuffer: A container data structure that stores video images before and after encoding and decoding.
The following figure is a schematic diagram of the data structure before and after H264 decoding:
2. How to use the hard solution:
The code stream of H264 is composed of NALU units, which contain video image data and H264 parameter information. Among them, the video image data is CMBlockBuffer, and the parameter information of H264 can be combined into FormatDesc. Specifically, parameter information includes SPS (Sequence Parameter Set) and PPS (Picture Parameter Set). The following figure shows the structure of an H264 stream:
Decoding method 1: (Decode and display through the AVSampleBufferDisplayLayer provided by the system)
1. Initialize the H264 hardware solution param:
1) Extract sps and pps to generate format description;
//sps
_spsSize =format.getCsd_0_size()-4;_sps = (uint8_t *)malloc(_spsSize);memcpy(_sps,format.getCsd_0()+4, _spsSize);
//pps
_ppsSize =format.getCsd_1_size()-4;_pps = (uint8_t *)malloc(_ppsSize);memcpy(_pps,format.getCsd_1()+4, _ppsSize);
2) Use the CMVideoFormatDescriptionCreateFromH264ParameterSets function to build CMVideoFormatDescriptionRef.
CMVideoFormatDescriptionCreateFromH264ParameterSets(kCFAllocatorDefault,
2, //param count parameterSetPointers, parameterSetSizes, 4, //nalstartcodesize&_decoderFormatDescription);
2. Convert the H264 code stream into CMSampleBuffer before decoding:
1) Use the CMBlockBufferCreateWithMemoryBlock interface to construct a CMBlockBufferRef;
CMBlockBufferRef blockBuffer=NULL;CMBlockBufferCreateWithMemoryBlock(kCFAllocatorDefault, (void*)frame.bytes, frame.length, kCFAllocatorNull,NULL,0, frame.length,0,&blockBuffer);
2) Obtain CMVideoFormatDescriptionRef, CMBlockBufferRef and optional time information according to the above, and use the CMSampleBufferCreate interface to obtain CMSampleBuffer data, the original data to be decoded.
CMSampleBufferRef sampleBuffer =NULL;CMSampleBufferCreateReady(kCFAllocatorDefault, blockBuffer, _decoderFormatDescription,1,0,NULL,1, sampleSizeArray, &sampleBuffer);
3. Hard solution image display:
It is decoded and displayed by the AVSampleBufferDisplayLayer provided by the system.
AVSampleBufferDisplayLayer is a display layer provided by Apple that specifically displays encoded H264 data. It is a subclass of CALayer, so its usage is similar to other CALayers. This layer has a built-in hardware decoding function, which can directly display the decoded image of the original CMSampleBuffer on the screen, which is very simple and convenient.
CFArrayRef attachments = CMSampleBufferGetSampleAttachmentsArray(sampleBuffer,YES);CFMutableDictionaryRef dict = (CFMutableDictionaryRef)CFArrayGetValueAtIndex(attachments,0);CFDictionarySetValue(dict, kCMSampleAttachmentKey_DisplayImmediately, kCFBooleanTrue);if(status == kCMBlockBufferNoErr) {
if([_avslayer isReadyForMoreMediaData]) {dispatch_sync(dispatch_get_main_queue(),^{
[_avslayer enqueueSampleBuffer:sampleBuffer];
});
}
CFRelease(sampleBuffer);
}
Decoding method 2: (Decode CMSampleBuffer into an image through the VTDecompression interface, and display the image through UIImageView or OpenGL)
1. Initialize the H264 hardware solution param:
On the basis of method 1, use the VTDecompressionSessionCreate interface to construct VTDecompressionSessionRef; (initialize VTDecompressionSession, set the relevant information of the decoder)
VTDecompressionSessionRef _deocderSession;VTDecompressionSessionCreate(kCFAllocatorDefault, _decoderFormatDescription,NULL, attrs, &callBackRecord, &_deocderSession);
2. Convert the H264 code stream into CMSampleBuffer before decoding:
same way one
3. Use the VTDecompressionSessionDecodeFrame interface to decode the CMSampleBuffer data into CVPixelBufferRef data:
CVPixelBufferRef outputPixelBuffer=NULL;
VTDecompressionSessionDecodeFrame(_deocderSession,
sampleBuffer,
flags,
&outputPixelBuffer,
&flagOut);
4. Convert CVPixelBufferRef data into UIImage and display:
CIImage*ciImage= [CIImage imageWithCVPixelBuffer:outputPixelBuffer];UIImage*uiImage= [UIImage imageWithCIImage:ciImage];
3. Program flow diagram:
Decoding method one
Decoding method two
4. Comparison of two decoding methods:
Decoding method one:
Advantages: This method decodes and displays through the AVSampleBufferDisplayLayer display layer provided by the system. This layer has a built-in hardware decoding function, which directly displays the decoded image of the original CMSampleBuffer on the screen, which is very simple and convenient, and has high execution efficiency and relatively small memory usage.
Disadvantages: Image data cannot be directly obtained from the decoded data and processed accordingly, and the decoded data cannot be directly used in other applications (generally more complex conversions are required).
Decoding method two:
Advantages: This method obtains CVPixelBufferRef data through the VTDecompressionSessionDecodeFrame interface. We can directly obtain image data from CVPixelBufferRef data and process it accordingly, which is convenient for other applications.
Disadvantages: The execution efficiency in decoding is relatively low, and the memory occupied will be relatively large.
').addClass('pre-numbering').hide(); $(this).addClass('has-numbering').parent().append($numbering); for (i = 1; i <= lines; i++) { $numbering.append($('
').text(i)); }; $numbering.fadeIn(1700); }); });
The above has introduced the H264 hard solution and display instructions in the iOS system, including aspects of the content, and I hope it will be helpful to friends who are interested in IOS development.
Add a demo address on personal github: h264 hard decoding demo
Author: Round Moon
Link: https://www.jianshu.com/p/fb78ee430949
Source: Jianshu
The copyright belongs to the author. For commercial reprint, please contact the author for authorization, for non-commercial reprint, please indicate the source.