Everyone must have known some blockbuster movies such as "Kung Fu Panda" and "Teenage Mutant Ninja Turtles". The modeling characters in them can perform human actions well, so how are these achieved? Today, I will work with you to capture the motion of a character in a video from scratch and add it to the modeled character, so that the modeled virtual character can also achieve a variety of behaviors like humans, The implementation of this project is a bit troublesome. Interested friends can try it step by step, and they will learn a lot after completion.

2. Project introduction

1. Purpose

The name of this project is 3D motion capture . It is not difficult to guess that it is necessary to capture the poses of moving characters in videos or cameras and display them in 3D. To put it simply, it is to allow the movements of the characters to be extracted to facilitate subsequent research and observation. It can also be used to create virtual videos and bring visual stimulation to the audience.

2. Technical points

(1) Read the file

Since it is called 3D motion capture, it is definitely necessary to capture the poses of the characters in the video or camera. The first step is of course to read the file. The read file only needs to be a complete character motion video, so if you shoot The video camera is fixed , which will greatly improve the accuracy of our project.

(2) Mark key nodes

After getting the video file, the next step is to mark the key nodes of the body of the characters in the video . This is a piece of cake for the third-party library cvzone in Python . The library records 33 key nodes of the human body. After reading the video, the key nodes of the characters can be marked. Through the change of node coordinates, we can grasp the movement posture of the characters.

(3) Read the coordinates of key nodes in each frame of image

We have obtained the coordinates of 33 key nodes , and their values will change in each frame of image, so we need to record the coordinate values in each frame of image . We can create a file to store the recorded coordinate values, and these coordinate points will be used in the subsequent process. I will explain the specific record implementation method in the following implementation steps, I hope you can bear your temper and continue to read! !

(4) Create 3D characters in Unity

Use Unity to create a simple character consisting of 33 spheres and several line segments . I won’t go into details about the download and installation of Unity here. You can search it online and search a lot! ! The steps of creating a 3D character will be described in detail in the implementation steps below.

(5) Make character movements 3D

Mainly use C# script in Unity to connect the change of our coordinate points to the created 3D character, the other is to adjust some parameters of the camera in Unity to make our 3D image more reasonable and 'photogenic' .

3. What to do

(1) Third-party library

In this project, we use Python mainly to capture the movement of people and mark key nodes, and record the changes of simple coordinate points, so we only need some basic vision libraries:

import cv2
from cvzone.PoseModule import PoseDetector

When using cvzone, I have encountered some problems and recorded them. If someone encounters the same problem, they should be able to help you easily: PoseDetector in cvzone reads video coordinate information problem

(2) Unity installation and use

There is no big problem with the download and installation of this software. I downloaded it using the vx public account Partner God , but there is a problem with the 2020 version it provides. Download and install the 2019 version. As for using it, let’s learn and sell it now, it’s not difficult anyway.

(3) C# understanding

Using C# language, we need to write scripts to connect coordinate points and 3D models. You may feel that you have never learned it, it is difficult, etc., but don’t be afraid, we don’t need to master everything, just use a few grammars, create a few A list is enough to achieve our goal.

3. Project Implementation Steps

1. Python part

Our project this time needs to be completed by combining multiple tools. Using Python, we can get the motion status of the characters in the video file and mark the 33 key nodes on the body , and then store the coordinate values of these key nodes in a file . .

(1) Read character actions and mark key nodes

After reading the video file, it is necessary to mark the key nodes on the body of the characters in the video. Here we only need to use the following function

img = detector.findPose(img)#mark pose key nodes

We can mark the coordinate points we need, as follows:

But we only marked the nodes, and did not record the coordinates of each node, so we need to create a list to store the coordinates of the key points in each frame of the image, so that we can store these points later Save it to a file, of course we can also print the length of the list we store to judge the number of frames of our video. We need to use the following function to help us read the coordinate value of each coordinate point:

lmList, bboxInfo = detector.findPosition(img) #Get the posture information in the video

With the above steps, we can detect the movement status of the characters in the video. The staged code is as follows (for reference only, the overall code will be posted later in the article):

cap = cv2.VideoCapture('Video.mp4')
detector = PoseDetector()
posList = []
while True:
    success, img = cap.read()

    img = detector.findPose(img)#标记姿势关键节点

    lmList, bboxInfo = detector.findPosition(img) #获取视频中的姿势信息

    if bboxInfo:            #判断是否检测到一个身体
        lmString = ''       #创建一个空的字符串
        for lm in lmList:   #'lm'为'landmark'的缩写，即33个关键节点

            lmString += f'{lm[1]},{img.shape[0] - lm[2]},{lm[3]},' #最后一个逗号保留，这样每一个坐标值才能分离开来
        posList.append(lmString)

    print(len(posList))#打印视频帧数

(2) Write the coordinate value into the file and save it

The above has stored our coordinate values in a list, but for subsequent 3D conversion, we need to store these coordinate values in a .txt file . code show as below:

    if key == ord('s'):   #将坐标点写入一个 .txt文件
     with open("AnimationFile.txt", 'w') as f:
        f.writelines(["%s\n" % item for item in posList])

When we press 's' at the end of the video, all the coordinate points we get will be stored in the 'AnimationFile.txt' file, and we can judge the coordinate point storage in our file by the number of frames obtained earlier.

For example, the number of frames we read is as follows:

Then the number of data lines in the obtained file is as follows:

A few lines less than the number of video frames, but acceptable because we can't press 's' at the last frame of the video either.

2. Unity part

After using Python to record the coordinate values of the key nodes of the characters, we need to apply the recorded coordinate points to the 3D model. Here we need to use Unity to create it. Generally speaking, it is relatively simple, just try boldly. .

(1) Create a project

First we need to create a new project, then right-click in the project to create a blank , named manager, this blank folder is more important, we need to use it to store some of our C# scripts later.

(2) Create blanks to store 3D structure parts

To complete a 3D modeling, we need some lines, spheres and other objects, and the blank folder we created is used to store these parts . As for the naming method of the folder, you can consider it yourself. Here I named it body , which holds our sphere and lines.

(3) Create a sphere

We need 33 spheres to represent the obtained 33 key nodes, so the next thing to do is to right- click on the body and create a sphere, right-click on the blank space below to create a material, select the color as your favorite color, and drag Move the newly created material to the sphere under the body, so that a sphere with a custom color can be created. If necessary, you can click on the sphere and modify the size in the expanded column on the right, and finally copy the sphere in the body and paste it Under the body, paste a total of 32 times . As for the name, you can consider it yourself.

(4) Connection of key nodes and models

We need to copy the key node coordinate value file obtained in the Python part to our project, and create a C# script to associate it. We need to drag this C# file to the manager file for management.

The above two pictures are the script file we created and the view added to the manager.

Next, we need to change the size of the body class to 33 and drag the spheres below into the corresponding objects in order, and then we write a C# script to associate. I will put the script code in the later part of the article.

(5) Connect the scattered spheres

Through the previous steps, we have been able to make the 33 points complete some simple actions, but we still try to connect those scattered points, so that our project is relatively complete.

First of all, we need to create a blank under the body and name it Lins (because we will use it to store all other lines under this blank, so the name is plural), and then follow the previous steps to create a sphere. We can create it under Lines A line and choose the appropriate material.

Next, we need to create a script again. The main purpose is to select the starting point and end point for each line (in fact, each previously created sphere). The code of the script will still be published later in the article. Through this script, we can achieve line alignment. position setting.

How to connect these lines? We can observe the human body nodes in mediapipe , combined with the following picture, we can set the start and end points of each line:

Then we copy the first line before, and paste it to Lines, and then do a simple and boring drag ( drag the corresponding sphere to the start and end of the line, one-to-one correspondence ):

Then we click Run, and we should be able to see that our project is successful.

Fourth, the overall code

1. Python code

"""
Author:XiaoMa
CSDN Address:一马归一码
"""
import cv2
from cvzone.PoseModule import PoseDetector

cap = cv2.VideoCapture('Video.mp4')
detector = PoseDetector()
posList = []
while True:
    success, img = cap.read()

    img = detector.findPose(img)#标记姿势关键节点

    lmList, bboxInfo = detector.findPosition(img) #获取视频中的姿势信息

    if bboxInfo:            #判断是否检测到一个身体
        lmString = ''       #创建一个空的字符串
        for lm in lmList:   #'lm'为'landmark'的缩写，即33个关键节点

            lmString += f'{lm[1]},{img.shape[0] - lm[2]},{lm[3]},' #最后一个逗号保留，这样每一个坐标值才能分离开来
        posList.append(lmString)

    print(len(posList))

    cv2.imshow("Image", img)
    key = cv2.waitKey(1)

    if key == ord('s'):   #将坐标点写入一个文件
     with open("AnimationFile.txt", 'w') as f:
        f.writelines(["%s\n" % item for item in posList])

2. C# script

(1) Sphere animation

using System.Collections;
using System.Collections.Generic;
using System.Linq;
using UnityEngine;
using System.Threading;

public class AnimationCode : MonoBehaviour
{

    public GameObject[] Body;
    List<string> lines;
    int counter = 0;

    void Start()
    {
        lines = System.IO.File.ReadLines("Assets/AnimationFile.txt").ToList();
    }


    void Update()
    {
        string[] points = lines[counter].Split(',');

        for (int i =0; i<=32;i++)
        {
            float x = float.Parse(points[0 + (i * 3)]) / 100;
            float y = float.Parse(points[1 + (i * 3)]) / 100;
            float z = float.Parse(points[2 + (i * 3)]) / 300;
            Body[i].transform.localPosition = new Vector3(x, y, z);
        }

  
        counter += 1;
        if (counter == lines.Count) { counter = 0; }
        Thread.Sleep(30);
    }
}

(2) Lines

using System.Collections;
using System.Collections.Generic;
using UnityEngine;

public class LineCode : MonoBehaviour
{

    LineRenderer lineRenderer;

    public Transform origin;
    public Transform destination;

    void Start()
    {
        lineRenderer = GetComponent<LineRenderer>();
        lineRenderer.startWidth = 0.1f;
        lineRenderer.endWidth = 0.1f;
    }

    void Update()
    {
        lineRenderer.SetPosition(0, origin.position);
        lineRenderer.SetPosition(1, destination.position);
    }
}

5. Conclusion

The realization of this project is not very difficult, especially the Python part, but it is more cumbersome to reproduce, and it should be no problem to follow it step by step. The tree of the embrace is born at the end of the hair, come on!

3D Motion Capture (3D Motion Capture)

I. Introduction