[UE Unreal Camera] [Nanny Level Tutorial 2] [Includes source code] Teach you how to obtain camera frame data through UE

  [UE Unreal Camera] [Nanny Level Tutorial 2] [Includes source code] Teach you how to obtain camera frame data through UE~
c6ebbaddb1aff.png)

overview

  In UE Camera Tutorial 1, we have opened the camera through the media player that comes with Unreal, and displayed the data of the camera on the game screen. Of course, this is only the most basic function. In general, we need to process the image data of the camera, such as face detection, or draw new content in the image, or just save the image data. What should we do at this time? ? Regarding this part of the content, there are actually very few online materials, and there are no specific instructions in the official documents, but don’t worry, I’ve already stepped on the pit for you, just follow me step by step~

method introduction

  First, sort out the logic, how do we turn on the camera and preview the screen?
  1. In order to open the camera , first open the camera through the MediaPlayer ( CameraMeidaPlayer ), when the MediaPlayer is created, the system will automatically create a MediaTexture ( CameraMeidaPlayer_Video ) for it. At this point, we can already see the MediaPlayer and MediaTexture in the UE editor. The camera image is shown.
  2. In order to display the picture , we dragged a Plane (plane) in the game level, and then dragged the MediaTexture to the Plane. At this time, the engine created a material ( CameraMediaPlayer_Video_Mat ) for us.
  This is what we did in the previous section. Take a look at our content browser, now we have CameraMeidaPlayer, CameraMediaPlayer_Video_Mat, CameraMeidaPlayer_Video three things.

  The core of obtaining frame data is how to convert CameraMediaPlayer_Video_Mat into frame data, which is an array containing image information in C++ code. The specific idea is to export the image information through the rendering target (RenderTarget2D).
  Simply put, a render target (Render Target) is a texture that can be written at runtime. From the engine's point of view, the render target stores information such as color, normal and AO.

Specific steps

  1. Create a RenderTarget2D, name it CameraRender2D here.

  2. Create a variable of RenderTarget2D in the blueprint interface, name it RenderTarget here, and point its default value to CameraRender2D in the content manager. 3. Create

  a variable of Material in the blueprint interface, name it CameraMaterial here, and then Set its default value to CameraMdiaPlayer_Video_Mat in the content manager.

  4. Start drawing the blueprint. The example in the figure is to click on the plane on the screen, every time you click, it will draw RenderTarget2D, after drawing, RenderTarget2D will save the image information, at this time, input it into ProcessData, and use C++ to process the image data Processing, after getting the result we want, we can release the resources through the rendering target.

  In c++, how to convert Target2D to image data? Through the following code, first we get the Target2D in the above steps, and input it as parameter rt in the function. First convert rt into an array of data format TArray in Unreal, and then convert this array into a more general c-form unsigned char format. In other words, we have obtained the data of the image and saved it in the unsinged char format pointer in the figure below! Note that the image data format is RGBA format, which is not difficult to see from the code. As for how to manipulate this image data, please feel free to play~

bool ProcessData(UTextureRenderTarget2D* rt) {
    
    
   //Get Imagedata in the format of TArray<FColor>
    if (rt == nullptr) {
    
    
        GEngine->AddOnScreenDebugMessage(INDEX_NONE, 20.0f, FColor::Yellow, TEXT("UTextureRenderTarget2D == nullptr,exit"));
        return false;
    }
    FTextureRenderTargetResource* rtResource = rt->GameThread_GetRenderTargetResource();
    FReadSurfaceDataFlags readPixelFlags(RCM_UNorm);
    readPixelFlags.SetLinearToGamma(true);

    TArray<FColor> outBMP;
    outBMP.AddUninitialized(rt->GetSurfaceWidth() * rt->GetSurfaceHeight());
    rtResource->ReadPixels(outBMP, readPixelFlags);


    for (FColor& color : outBMP)
        color.A = 255;
    //Convert the image data format from TArray to const char*(c style)
 unsigned long dataSize = sizeof(unsigned char) * 4 * (rt->GetSurfaceWidth() * rt->GetSurfaceHeight());
    unsigned char *imgData = (unsigned char *)malloc(dataSize);
    memset(imgData, 0, dataSize);
    int cusPos = 0;
    for (FColor& color : outBMP) {
    
    
        memcpy(imgData + cusPos, (unsigned char*)&color.R, 1);
        cusPos += 1;
        memcpy(imgData + cusPos, (unsigned char*)&color.G, 1);
        cusPos += 1;
        memcpy(imgData + cusPos, (unsigned char*)&color.B, 1);
        cusPos += 1;
        memcpy(imgData + cusPos, (unsigned char*)&color.A, 1);
        cusPos += 1;
    }
 }

  Have you done this step? If your RenderTarget already has an image, as shown in the picture below, then congratulations, you have completed the above steps correctly.

method introduction

Demo project description

  The Demo project contains examples of the content mentioned above. After entering the project, click Run, and you can see that the Windows computer camera is turned on and displayed in the game. After packaging, it can also be used on Android phones. (Note that Mac computers and iOS phones are not available)
  Follow the official account and send: Unreal-Camera.