Prediction of 3D points in ThreeDPoseTracker

    private IEnumerator ExecuteModelAsync()
    {
        if(Lock)
        {
            yield return null;
        }

        // Create input and Execute model
        yield return _worker.StartManualSchedule(inputs);

        if (!Lock)
        {
            // Get outputs
            for (var i = 2; i < _model.outputs.Count; i++)
            {
                b_outputs[i] = _worker.PeekOutput(_model.outputs[i]);
            }

            // Get data from outputs
            //heatMap2D = b_outputs[0].data.Download(b_outputs[0].shape);
            //offset2D = b_outputs[1].data.Download(b_outputs[1].shape);
            offset3D = b_outputs[2].data.Download(b_outputs[2].shape);
            heatMap3D = b_outputs[3].data.Download(b_outputs[3].shape);

            PredictPose();

            fpsCounter++;
        }
    }

  

Called in the Update() loop:

UpdateVNectModel();

And UpdateVNectModel = new UpdateVNectModelDelegate(UpdateVNectAsync); The delegate calls UpdateVNectAsync

The coroutine ExecuteModelAsync is enabled in UpdateVNectAsync,

This coroutine uses the yield return statement to return a result of type IEnumerable. The yield return statement does not terminate the execution of the method, but suspends the execution of the method and returns the current value to the caller. When the caller requests the next value again, the method continues execution from where it left off until it encounters the next yield return statement or the end of the method.

_worker.StartManualSchedule(inputs) does model reasoning, and inputs are a set of video pictures

 _worker.StartManualSchedule, after the completion, the output will be stored in the _worker, and the output will be taken out by calling _worker.PeekOutput and placed in b_outputs

 According to the shape of each output, download its data into the corresponding float array (offset3D and heatMap3D).

Call the PredictPose method, calculate the three-dimensional coordinates of each joint point according to the values ​​in the offset3D and heatMap3D arrays, and store them in the jointPoints array.

Finally, add one to the fpsCounter variable, indicating that the model has been executed once. This variable can be used to count the execution frequency of the model.

------------------------------------------------------------------------------------------------------------------------------

The function of the PredictPose method is to predict the three-dimensional pose of the human body based on the three-dimensional offset and heat map output by the model, and filter and smooth the position of each joint point. Specifically, it does the following steps:

• First, some variables are defined, such as score, filterWindow, cubeOffsetSquared, etc., for subsequent calculations.
• Then, use a loop to traverse each joint point (JointNum = 19), and find the maximum probability position (maxXIndex, maxYIndex, maxZIndex) of each joint point in the 3D heat map. Then according to the three-dimensional offset array (offset3D) and the size of the heat map (HeatMapCol = 32), calculate the coordinates of each joint point in three-dimensional space (jp.Now3D.x, jp.Now3D.y, jp.Now3D. z), and add it to the filter. At the same time, the probability value (jp.Score3D) of each joint point is accumulated and used to calculate the average score (EstimatedScore).
• Then, according to some known joint point positions, calculate some unknown joint point positions, such as hip, neck, head and spine. Some simple geometric methods are used here, such as taking the midpoint of two points or calculating the normal vector of a triangle (TriangleNormal).
• Finally, the position of each joint point is filtered and smoothed using a Kalman filter (KalmanUpdate) and a low-pass filter (NOrderLPF), and the visibility (jp.Visibled) of each joint point is updated. Then, according to the orientation angle of the human body (frwdAngle), it is judged whether the human body pose model (VNectModel.IsPoseUpdate) needs to be updated, and the corresponding thresholds (ForwardThreshold and BackwardThreshold) are set.