UE ReplayKit for iOS插件使用

iOSReplayKit用途

ReplayKit是苹果为iOS/tvOS/macOS平台视频直播和视频录制提供的工具包（Record or stream video from the screen, and audio from the app and microphone.），整个插件使用很简单（因为大部分功能都被苹果限制死了），调用StartXxxxWithHandler开始录制，提供回调函数接收调用结果和数据包；StopXxxxWithHandler结束录制，提供回调函数接收调用结果。

iOSReplayKit（ReplayKit for iOS）插件是提供给UE使用iOS平台ReplayKit软件包的桥接器。插件包含一个蓝图类UIOSReplayKitControl，提供调用iOS ReplayKit API函数的蓝图接口；一个Objective-C语言写的ReplayKitRecorder实现类。

开始使用

因为要在Unreal开发的iOS平台APP使用视频录制功能，查找资料发现苹果提供的ReplayKit可以支持视频和麦克风录制。进一步查找Unreal平台插件，在github上找到一个replaykit for ios的项目
PushkinStudio/PsReplayKit
但这个项目最后更新已经是2018年了。
后来查看Unreal引擎源码，发现Unreal引擎里面已经集成ReplayKit插件。
直接在项目引入插件，重启
插件使用很简单，一共提供了四个函数

Unreal本着已经提供了源代码没必要再提供文档的原则，查看帮助文档，得到的信息也不会比函数名更多了。

至于StartRecording和StartCaptureToFile有什么区别，查看源代码，发现确实是Apple的锅，Apple ReplayKit提供startRecordingWithHandler和startCaptureWithHandler两个方法，startCaptureWithHandler提供了回调函数，可以自己处理录制过程；startRecordingWithHandler不提供过程回调，仅在stop时提供预览（调用系统预览框），全程开发者无法干预。

创建两个按钮，点击绑定调用StartCaptureToFile和StopCapture，打包输出，在真机上运行点击Start录制，成功唤起权限确认对话框。

点击Stop结束，显示存储到相册权限提示，看起来一切正常。
在测试几次，结果发现问题了，除了一开始成功的一次，后面基本上都是失败的，并没有任何文件存储到相册。
然后换了StartRecording和StopRecording接口，一样的问题，偶尔成功，大部分失败。
查看源代码，发现StartCaptureToFile调用startCapture，该函数实现代码是自己将回调数据包写入mp4文件，其中有处理文件的代码

  // create the asset writer
  [_assetWriter release] ;
  // todo: do we care about the file name? support cleaning up old captures?
  auto fileName = FString::Printf(TEXT("%s.mp4"), *FGuid::NewGuid().ToString());
  _captureFilePath = [captureDir stringByAppendingFormat : @"/%@", fileName.GetNSString()];
  [_captureFilePath retain] ;

  _assetWriter = [[AVAssetWriter alloc]initWithURL: [NSURL fileURLWithPath : _captureFilePath] fileType : AVFileTypeMPEG4 error : nil];
  // _assetWriter = [AVAssetWriter assetWriterWithURL : [NSURL fileURLWithPath : _captureFilePath] fileType : AVFileTypeMPEG4 error : nil];
  [_assetWriter retain] ;

通过USB连接iPAD，在文件浏览里面发现Capture目录下大部分录制的mp4都是0字节文件，说明在写入环节出现问题了

查看写入代码，增加Unreal屏幕打印调式信息功能

        if (_assetWriter.status == AVAssetWriterStatusUnknown)
        {
    
    
          [_assetWriter startWriting] ;
          [_assetWriter startSessionAtSourceTime : CMSampleBufferGetPresentationTimeStamp(sampleBuffer)] ;
          if (GEngine) {
    
    
            CMTime time = CMSampleBufferGetPresentationTimeStamp(sampleBuffer);
            GEngine->AddOnScreenDebugMessage(-1, 5.f, FColor::Blue, FString::Printf(TEXT("_assetWriter startWriting: %d"), (int)CMTimeGetSeconds(time)));
          }
        }
        if (_assetWriter.status == AVAssetWriterStatusFailed)
        {
    
    
          NSLog(@"%d: %@", (int)_assetWriter.error.code, _assetWriter.error.localizedDescription);
          if (GEngine) {
    
    
            const char* dest = [_assetWriter.error.localizedDescription UTF8String];
            GEngine->AddOnScreenDebugMessage(-1, 5.f, FColor::Green, FString::Printf(TEXT("_assetWriter failed: %d %s"), (int)_assetWriter.error.code, UTF8_TO_TCHAR(dest)));
          }
        }

运行后屏幕打印的调试信息表明回调函数没有问题，问题出在AVAssetWriter状态是AVAssetWriterStatusFailed，错误代码为-11823，错误描述cannot save。
经历漫长编译-输出-安装-调试，发现蓝色提示信息_assetWriter startWriting 16600

GEngine->AddOnScreenDebugMessage(-1, 5.f, FColor::Blue, FString::Printf(TEXT("_assetWriter startWriting: %d"), (int)CMTimeGetSeconds(time)));

夹杂在绿色-11823，cannot save错误信息之间显示了两条。蓝色提示信息是在调用_assetWriter.startWriting时显示的，显然调用了两次startWriting导致AV视频写入器报错了。

        if (_assetWriter.status == AVAssetWriterStatusUnknown)
        {
    
    
          [_assetWriter startWriting] ;
          // ...
        }

代码在调用startWriting时判断了写入器状态是不是无状态，调用startWriting以后写入器的状态应该会发生变化，线性调用是不会导致调用两次的，除非是多线程同时调用。连续调用了两次startWriting说明在状态改变之前又有新的回调函数调用了，猜想视频数据包和音频数据包处理是在不同线程进行的，几乎同时调用了回调函数，而回调函数并没有采用任何的线程锁机制，导致startWriting被调用了两次，从而引起AVAssetWriterStatusFailed错误，而偶尔成功的几次，应该是多线程没有同时回调，侥幸成功。

修改代码

增加一个frames自增变量，仅在0==frames时调用startWriting函数

        if (0 == _frames++) {
    
    
          [_assetWriter startWriting] ;
          [_assetWriter startSessionAtSourceTime : CMSampleBufferGetPresentationTimeStamp(sampleBuffer)] ;
          if (GEngine) {
    
    
            CMTime time = CMSampleBufferGetPresentationTimeStamp(sampleBuffer);
            GEngine->AddOnScreenDebugMessage(-1, 5.f, FColor::Blue, FString::Printf(TEXT("_assetWriter startWriting: %d"), (int)CMTimeGetSeconds(time)));
          }
        }

这里没有用线程锁，用了++自增操作，++操作仅一个CPU指令周期，几乎不可能被中断。

增加MIC录制

          AVAssetWriterInput* input = nullptr;
          // ...
          else if (bufferType == RPSampleBufferTypeAudioMic)
          {
    
    
            // todo?
          }
          // ...

bufferType == RPSampleBufferTypeAudioMic表示这是一个Microphone采集回调数据，但代码中仅有一个todo?注释，并没有任何代码实现，也就是直接使用插件是不能录制麦克风声音的。（直到5.1版本的iOSReplayKit都没有实现Mic采集）
仿照RPSampleBufferTypeAudioApp内录音频，创建一个Mic录制写入器

  // create the audio input
  [_audioInput release] ;
  AudioChannelLayout acl;
  bzero(&acl, sizeof(acl));
  acl.mChannelLayoutTag = kAudioChannelLayoutTag_Stereo;
  auto audioSettings = @{
    
    
    AVFormatIDKey: @(kAudioFormatMPEG4AAC),
    AVSampleRateKey: @(44100),
    AVChannelLayoutKey: [NSData dataWithBytes : &acl length : sizeof(acl)] ,
  };
  _audioInput = [AVAssetWriterInput assetWriterInputWithMediaType : AVMediaTypeAudio outputSettings : audioSettings];
  _audioInput.expectsMediaDataInRealTime = YES;
  [_audioInput retain] ;

  // create the microphone input
  [_microInput release] ;
  AudioChannelLayout mic;
  bzero(&mic, sizeof(mic));
  mic.mChannelLayoutTag = kAudioChannelLayoutTag_Mono;
  auto micSettings = @{
    
    
    AVFormatIDKey: @(kAudioFormatMPEG4AAC),
    AVSampleRateKey: @(44100),
    AVChannelLayoutKey: [NSData dataWithBytes : &mic length : sizeof(mic)] ,
  };
  _microInput = [AVAssetWriterInput assetWriterInputWithMediaType : AVMediaTypeAudio outputSettings : micSettings];
  _microInput.expectsMediaDataInRealTime = YES;
  [_microInput retain] ;

  // add the input to the writer
  // ...

  if ([_assetWriter canAddInput : _audioInput])
  {
    
    
    [_assetWriter addInput : _audioInput] ;
  }

  if ([_assetWriter canAddInput : _microInput])
  {
    
    
    [_assetWriter addInput : _microInput] ;
  }

todo?注释增加mic input输入处理

          AVAssetWriterInput* input = nullptr;
          if (bufferType == RPSampleBufferTypeVideo)
          {
    
    
            input = _videoInput;
          }
          else if (bufferType == RPSampleBufferTypeAudioApp)
          {
    
    
            input = _audioInput;
          }
          else if (bufferType == RPSampleBufferTypeAudioMic)
          {
    
    
            // todo?
            input = _microInput;
          }

          if (input && input.isReadyForMoreMediaData)
          {
    
    
            [input appendSampleBuffer : sampleBuffer] ;
          }

增加上面代码后，启动startCapture时勾选麦克风，就可以录制麦克风声音了（不知道官方插件为啥不实现这个简单的功能）

完整代码

// Copyright Epic Games, Inc. All Rights Reserved.

#include "ReplayKitRecorder.h"



#if PLATFORM_IOS
#include "IOSAppDelegate.h"
#include "IOS/IOSView.h"
#include "Misc/Paths.h"
#include "Engine/Engine.h"
#include "Engine/GameViewportClient.h"

@implementation ReplayKitRecorder

RPScreenRecorder* _Nullable _screenRecorder;
RPBroadcastActivityViewController* _Nullable _broadcastActivityController;
RPBroadcastController* _Nullable _broadcastController;

// stuff used when capturing to file
AVAssetWriter* _Nullable _assetWriter;
AVAssetWriterInput* _Nullable _videoInput;
AVAssetWriterInput* _Nullable _audioInput;
AVAssetWriterInput* _Nullable _microInput;
NSString* _Nullable _captureFilePath;
int _frames;

- (void) initializeWithMicrophoneEnabled:(BOOL)bMicrophoneEnabled withCameraEnabled:(BOOL)bCameraEnabled {
    
    
	_screenRecorder = [RPScreenRecorder sharedRecorder];
	[_screenRecorder setDelegate:self];
#if !PLATFORM_TVOS
	[_screenRecorder setMicrophoneEnabled:bMicrophoneEnabled];
	[_screenRecorder setCameraEnabled:bCameraEnabled];
#endif
}

- (void)startRecording {
    
    
	// NOTE(omar): stop any live broadcasts before staring a local recording
	if( _broadcastController != nil ) {
    
    
		[self stopBroadcast];
	}
	
	if( [_screenRecorder isAvailable] ) {
    
    
		[_screenRecorder startRecordingWithHandler:^(NSError * _Nullable error) {
    
    
			if( error ) {
    
    
				NSLog( @"error starting screen recording");
			}
		}];
	}
}

- (void)stopRecording {
    
    
	if( [_screenRecorder isAvailable] && [_screenRecorder isRecording] ) {
    
    
		[_screenRecorder stopRecordingWithHandler:^(RPPreviewViewController * _Nullable previewViewController, NSError * _Nullable error) {
    
    
			[previewViewController setPreviewControllerDelegate:self];
		 
			// automatically show the video preview when recording is stopped
			previewViewController.popoverPresentationController.sourceView = (UIView* _Nullable)[IOSAppDelegate GetDelegate].IOSView;
			[[IOSAppDelegate GetDelegate].IOSController presentViewController:previewViewController animated:YES completion:nil];
		}];
	}
}

- (void)createCaptureContext
{
    
    
  auto fileManager = [NSFileManager defaultManager];
  auto docDir = [NSSearchPathForDirectoriesInDomains(NSDocumentDirectory, NSUserDomainMask, YES) objectAtIndex:0];
  auto captureDir = [docDir stringByAppendingFormat : @"/Captures"];
  [fileManager createDirectoryAtPath : captureDir withIntermediateDirectories : YES attributes : nil error : nil] ;

  [_captureFilePath release] ;

  // create the asset writer
  [_assetWriter release] ;
  // todo: do we care about the file name? support cleaning up old captures?
  auto fileName = FString::Printf(TEXT("%s.mp4"), *FGuid::NewGuid().ToString());
  _captureFilePath = [captureDir stringByAppendingFormat : @"/%@", fileName.GetNSString()];
  [_captureFilePath retain] ;

  _assetWriter = [[AVAssetWriter alloc]initWithURL: [NSURL fileURLWithPath : _captureFilePath] fileType : AVFileTypeMPEG4 error : nil];
  // _assetWriter = [AVAssetWriter assetWriterWithURL : [NSURL fileURLWithPath : _captureFilePath] fileType : AVFileTypeMPEG4 error : nil];
  [_assetWriter retain] ;

  // create the video input
  [_videoInput release] ;

  auto view = [IOSAppDelegate GetDelegate].IOSView;
  auto width = [NSNumber numberWithFloat : view.frame.size.width];
  auto height = [NSNumber numberWithFloat : view.frame.size.height];

  if (GEngine && GEngine->GameViewport && GEngine->GameViewport->Viewport)
  {
    
    
    auto viewportSize = GEngine->GameViewport->Viewport->GetSizeXY();
    width = [NSNumber numberWithInt : viewportSize.X];
    height = [NSNumber numberWithInt : viewportSize.Y];
  }

  auto videoSettings = @{
    
    
  AVVideoCodecKey: AVVideoCodecTypeH264,
    AVVideoWidthKey : width,
    AVVideoHeightKey : height
  };

  _videoInput = [AVAssetWriterInput assetWriterInputWithMediaType : AVMediaTypeVideo outputSettings : videoSettings];
  _videoInput.expectsMediaDataInRealTime = YES;
  [_videoInput retain] ;

  // create the audio input
  [_audioInput release] ;
  AudioChannelLayout acl;
  bzero(&acl, sizeof(acl));
  acl.mChannelLayoutTag = kAudioChannelLayoutTag_Stereo;
  auto audioSettings = @{
    
    
    AVFormatIDKey: @(kAudioFormatMPEG4AAC),
    AVSampleRateKey: @(44100),
    AVChannelLayoutKey: [NSData dataWithBytes : &acl length : sizeof(acl)] ,
  };
  _audioInput = [AVAssetWriterInput assetWriterInputWithMediaType : AVMediaTypeAudio outputSettings : audioSettings];
  _audioInput.expectsMediaDataInRealTime = YES;
  [_audioInput retain] ;

  // create the microphone input
  [_microInput release] ;
  AudioChannelLayout mic;
  bzero(&mic, sizeof(mic));
  mic.mChannelLayoutTag = kAudioChannelLayoutTag_Mono;
  auto micSettings = @{
    
    
    AVFormatIDKey: @(kAudioFormatMPEG4AAC),
    AVSampleRateKey: @(44100),
    AVChannelLayoutKey: [NSData dataWithBytes : &mic length : sizeof(mic)] ,
  };
  _microInput = [AVAssetWriterInput assetWriterInputWithMediaType : AVMediaTypeAudio outputSettings : micSettings];
  _microInput.expectsMediaDataInRealTime = YES;
  [_microInput retain] ;

  // add the input to the writer
  if ([_assetWriter canAddInput : _videoInput])
  {
    
    
    [_assetWriter addInput : _videoInput] ;
  }

  if ([_assetWriter canAddInput : _audioInput])
  {
    
    
    [_assetWriter addInput : _audioInput] ;
  }

  if ([_assetWriter canAddInput : _microInput])
  {
    
    
    [_assetWriter addInput : _microInput] ;
  }
}

- (void)startCapture
{
    
    
  if (_broadcastController)
  {
    
    
    [self stopBroadcast] ;
  }

  if ([_screenRecorder isAvailable])
  {
    
    
    _frames = 0;
    [self createCaptureContext] ;

    [_screenRecorder startCaptureWithHandler : ^ (CMSampleBufferRef sampleBuffer, RPSampleBufferType bufferType, NSError * error)
    {
    
    
      if (CMSampleBufferDataIsReady(sampleBuffer))
      {
    
    
        if (0 == _frames++) {
    
    
          [_assetWriter startWriting] ;
          [_assetWriter startSessionAtSourceTime : CMSampleBufferGetPresentationTimeStamp(sampleBuffer)] ;
          if (GEngine) {
    
    
            CMTime time = CMSampleBufferGetPresentationTimeStamp(sampleBuffer);
            GEngine->AddOnScreenDebugMessage(-1, 5.f, FColor::Blue, FString::Printf(TEXT("_assetWriter startWriting: %d"), (int)CMTimeGetSeconds(time)));
          }
        }
        /*
        if (_assetWriter.status == AVAssetWriterStatusUnknown)
        {
          [_assetWriter startWriting] ;
          [_assetWriter startSessionAtSourceTime : CMSampleBufferGetPresentationTimeStamp(sampleBuffer)] ;
          if (GEngine) {
            CMTime time = CMSampleBufferGetPresentationTimeStamp(sampleBuffer);
            GEngine->AddOnScreenDebugMessage(-1, 5.f, FColor::Blue, FString::Printf(TEXT("_assetWriter startWriting: %d"), (int)CMTimeGetSeconds(time)));
          }
        }
        */

        if (_assetWriter.status == AVAssetWriterStatusFailed)
        {
    
    
          // NSLog(@"%d: %@", (int)_assetWriter.error.code, _assetWriter.error.localizedDescription);
          if (GEngine) {
    
    
            const char* dest = [_assetWriter.error.localizedDescription UTF8String];
            GEngine->AddOnScreenDebugMessage(-1, 5.f, FColor::Green, FString::Printf(TEXT("_assetWriter failed: %d %s"), (int)_assetWriter.error.code, UTF8_TO_TCHAR(dest)));
          }
        }
        else {
    
    
          AVAssetWriterInput* input = nullptr;
          if (bufferType == RPSampleBufferTypeVideo)
          {
    
    
            input = _videoInput;
          }
          else if (bufferType == RPSampleBufferTypeAudioApp)
          {
    
    
            input = _audioInput;
          }
          else if (bufferType == RPSampleBufferTypeAudioMic)
          {
    
    
            // todo?
            input = _microInput;
          }

          if (input && input.isReadyForMoreMediaData)
          {
    
    
            [input appendSampleBuffer : sampleBuffer] ;
          }
        }
      }
    }

  completionHandler: ^ (NSError * error)
  {
    
    
    if (error)
    {
    
    
      NSLog(@"completionHandler: %@", error);
      if (GEngine) {
    
    
        GEngine->AddOnScreenDebugMessage(-1, 5.f, FColor::Red, FString::Printf(TEXT("error at startCapture")));
      }
    }
  }];
  }
}

- (void)stopCapture
{
    
    
    if ([_screenRecorder isAvailable])
    {
    
    
        [_screenRecorder stopCaptureWithHandler:^(NSError *error)
        {
    
    
            if (error)
            {
    
    
                NSLog(@"stopCaptureWithHandler: %@", error);
            }
            
            if (_assetWriter)
            {
    
    
                [_assetWriter finishWritingWithCompletionHandler:^()
                {
    
    
                    NSLog(@"finishWritingWithCompletionHandler");
                    
#if !PLATFORM_TVOS
                    if (UIVideoAtPathIsCompatibleWithSavedPhotosAlbum(_captureFilePath))
                    {
    
    
                        UISaveVideoAtPathToSavedPhotosAlbum(_captureFilePath, nil, nil, nil);
                        NSLog(@"capture saved to album");
                    }
#endif

                    [_captureFilePath release];
                    _captureFilePath = nullptr;
                    
                    [_videoInput release];
                    _videoInput = nullptr;
                    
                    [_audioInput release];
                    _audioInput = nullptr;

                    [_microInput release];
                    _microInput = nullptr;
                    
                    [_assetWriter release];
                    _assetWriter = nullptr;
                }];
            }
        }];
    }
}

//
// livestreaming functionality
//

- (void)startBroadcast {
    
    
	// NOTE(omar): ending any local recordings that might be active before starting a broadcast
	if( [_screenRecorder isRecording] ) {
    
    
		[self stopRecording];
	}
	
	[RPBroadcastActivityViewController loadBroadcastActivityViewControllerWithHandler:^(RPBroadcastActivityViewController * _Nullable broadcastActivityViewController, NSError * _Nullable error) {
    
    
		_broadcastActivityController = broadcastActivityViewController;
		[_broadcastActivityController setDelegate:self];
		[[IOSAppDelegate GetDelegate].IOSController presentViewController:_broadcastActivityController animated:YES completion:nil];
	}];
}

- (void)pauseBroadcast {
    
    
	if( [_broadcastController isBroadcasting] ) {
    
    
		[_broadcastController pauseBroadcast];
	}
}

- (void)resumeBroadcast {
    
    
	if( [_broadcastController isPaused] ) {
    
    
		[_broadcastController resumeBroadcast];
	}
}

- (void)stopBroadcast {
    
    
	if( [_broadcastController isBroadcasting ] ) {
    
    
		[_broadcastController finishBroadcastWithHandler:^(NSError * _Nullable error) {
    
    
			if( error ) {
    
    
				NSLog( @"error finishing broadcast" );
			}
			
			[_broadcastController release];
			_broadcastController = nil;
		}];
	}
}

//
// delegates
//

// screen recorder delegate
- (void)screenRecorder:(RPScreenRecorder* _Nullable)screenRecorder didStopRecordingWithError:(NSError* _Nullable)error previewViewController:(RPPreviewViewController* _Nullable)previewViewController {
    
    
	NSLog(@"RTRScreenRecorderDelegate::didStopRecrodingWithError");
	[previewViewController dismissViewControllerAnimated:YES completion:nil];
}

- (void)screenRecorderDidChangeAvailability:(RPScreenRecorder* _Nullable)screenRecorder {
    
    
	NSLog(@"RTRScreenRecorderDelegate::screenRecorderDidChangeAvailability");
}

// screen recorder preview view controller delegate
- (void)previewControllerDidFinish:(RPPreviewViewController* _Nullable)previewController {
    
    
	NSLog( @"RTRPreviewViewControllerDelegate::previewControllerDidFinish" );
	[previewController dismissViewControllerAnimated:YES completion:nil];
}

- (void)previewController:(RPPreviewViewController* _Nullable)previewController didFinishWithActivityTypes:(NSSet <NSString*> * _Nullable)activityTypes __TVOS_PROHIBITED {
    
    
	NSLog( @"RTRPreviewViewControllerDelegate::didFinishWithActivityTypes" );
	[previewController dismissViewControllerAnimated:YES completion:nil];
}

// broadcast activity view controller delegate
- (void)broadcastActivityViewController:(RPBroadcastActivityViewController* _Nullable)broadcastActivityViewController didFinishWithBroadcastController:(RPBroadcastController* _Nullable)broadcastController error:(NSError* _Nullable)error {
    
    
	NSLog( @"RPBroadcastActivityViewControllerDelegate::didFinishWithBroadcastController" );
	
	[broadcastActivityViewController dismissViewControllerAnimated:YES completion:^{
    
    
		_broadcastController = [broadcastController retain];
		[_broadcastController setDelegate:self];
		[_broadcastController startBroadcastWithHandler:^(NSError* _Nullable _error) {
    
    
			if( _error ) {
    
    
				NSLog( @"error starting broadcast" );
			}
		}];
	}];
}

// broadcast controller delegate
- (void)broadcastController:(RPBroadcastController* _Nullable)broadcastController didFinishWithError:(NSError* _Nullable)error {
    
    
	NSLog( @"RPBroadcastControllerDelegate::didFinishWithError" );
}

- (void)broadcastController:(RPBroadcastController* _Nullable)broadcastController didUpdateServiceInfo:(NSDictionary <NSString*, NSObject <NSCoding>*> *_Nullable)serviceInfo {
    
    
	NSLog( @"RPBroadcastControllerDelegate::didUpdateServiceInfo" );
}


@end

#endif

Objective-C和C++混合编码

Objective-C和C++虽然都是C语言，但他们语法差异还是很大的。Objective-C和C++混合编程很简单，将Objective-C代码放在.h和.cpp文件中，就可以直接写Objective-C代码，也可以直接写C++代码，不需要做任何特殊处理，编译器可以直接编译混合了Objective-C代码和C++代码的.h文件和.cpp文件。