Human body posture detection is realized by Opencv+Openpose

By chance, I learned about the calculation of human body posture. After learning K210, I thought about realizing this function through opencv. After searching a lot of information, I found that it can be realized by using opencv+openpose. Then I started to find some information. Deploy on pycharm.

foreword

An interesting application of human pose estimation is in CGI (computer graphic image, a filmmaking technique) application. If human poses can be detected, then graphics, styles, special effects enhancements, equipment, and artistic modeling can be loaded onto the human body. By tracking changes in human body posture, the rendered graphics can "naturally" "blend" with people when they move. An interesting application of pose estimation is tracking the motion of human objects in interactive games. The more popular Kinect uses 3D pose estimation (using IR sensor data) to track the movement of the human player, which can be used to render the movements of virtual characters.
Applications:
For detecting a person's fall or illness
For automatic teaching of fitness, sports and dance, etc.
For understanding body language of the whole body (such as airport runway signals, traffic police signals, etc.)
For enhanced security and monitoring

1. Environment configuration

pycharm2021.2.2
pycharm is a very useful software. At the beginning, we must configure the corresponding environment. Of course, you can also use the model training environment on my homepage. When you run it, the system will prompt you what environment is missing , and let you install it, you can install it directly. I won't go into too much detail here.

2. Use steps

1. Import file

After importing the corresponding file on pycharm, you can directly click to run, the system will prompt you what environment is missing, install what is missing, and install it through the terminal using pip.
Just leave an email if you need it, and I will send it to you.

2. Specific code

# To use Inference Engine backend, specify location of plugins:
# export LD_LIBRARY_PATH=/opt/intel/deeplearning_deploymenttoolkit/deployment_tools/external/mklml_lnx/lib:$LD_LIBRARY_PATH
import cv2 as cv
import numpy as np
import argparse

parser = argparse.ArgumentParser()
parser.add_argument('--input', help='Path to image or video. Skip to capture frames from camera')
parser.add_argument('--thr', default=0.2, type=float, help='Threshold value for pose parts heat map')
parser.add_argument('--width', default=368, type=int, help='Resize input to specific width.')
parser.add_argument('--height', default=368, type=int, help='Resize input to specific height.')

args = parser.parse_args()

BODY_PARTS = {
    
     "Nose": 0, "Neck": 1, "RShoulder": 2, "RElbow": 3, "RWrist": 4,
               "LShoulder": 5, "LElbow": 6, "LWrist": 7, "RHip": 8, "RKnee": 9,
               "RAnkle": 10, "LHip": 11, "LKnee": 12, "LAnkle": 13, "REye": 14,
               "LEye": 15, "REar": 16, "LEar": 17, "Background": 18 }

POSE_PAIRS = [ ["Neck", "RShoulder"], ["Neck", "LShoulder"], ["RShoulder", "RElbow"],
               ["RElbow", "RWrist"], ["LShoulder", "LElbow"], ["LElbow", "LWrist"],
               ["Neck", "RHip"], ["RHip", "RKnee"], ["RKnee", "RAnkle"], ["Neck", "LHip"],
               ["LHip", "LKnee"], ["LKnee", "LAnkle"], ["Neck", "Nose"], ["Nose", "REye"],
               ["REye", "REar"], ["Nose", "LEye"], ["LEye", "LEar"] ]

inWidth = args.width
inHeight = args.height

net = cv.dnn.readNetFromTensorflow("graph_opt.pb")

cap = cv.VideoCapture(args.input if args.input else 0)

while cv.waitKey(1) < 0:
    hasFrame, frame = cap.read()
    if not hasFrame:
        cv.waitKey()
        break

    frameWidth = frame.shape[1]
    frameHeight = frame.shape[0]
    
    net.setInput(cv.dnn.blobFromImage(frame, 1.0, (inWidth, inHeight), (127.5, 127.5, 127.5), swapRB=True, crop=False))
    out = net.forward()
    out = out[:, :19, :, :]  # MobileNet output [1, 57, -1, -1], we only need the first 19 elements

    assert(len(BODY_PARTS) == out.shape[1])

    points = []
    for i in range(len(BODY_PARTS)):
        # Slice heatmap of corresponging body's part.
        heatMap = out[0, i, :, :]

        # Originally, we try to find all the local maximums. To simplify a sample
        # we just find a global one. However only a single pose at the same time
        # could be detected this way.
        _, conf, _, point = cv.minMaxLoc(heatMap)
        x = (frameWidth * point[0]) / out.shape[3]
        y = (frameHeight * point[1]) / out.shape[2]
        # Add a point if it's confidence is higher than threshold.
        points.append((int(x), int(y)) if conf > args.thr else None)

    for pair in POSE_PAIRS:
        partFrom = pair[0]
        partTo = pair[1]
        assert(partFrom in BODY_PARTS)
        assert(partTo in BODY_PARTS)

        idFrom = BODY_PARTS[partFrom]
        idTo = BODY_PARTS[partTo]

        if points[idFrom] and points[idTo]:
            cv.line(frame, points[idFrom], points[idTo], (0, 255, 0), 3)
            cv.ellipse(frame, points[idFrom], (3, 3), 0, 0, 360, (0, 0, 255), cv.FILLED)
            cv.ellipse(frame, points[idTo], (3, 3), 0, 0, 360, (0, 0, 255), cv.FILLED)

    t, _ = net.getPerfProfile()
    freq = cv.getTickFrequency() / 1000
    cv.putText(frame, '%.2fms' % (t / freq), (10, 20), cv.FONT_HERSHEY_SIMPLEX, 0.5, (0, 0, 0))

    cv.imshow('OpenPose using OpenCV', frame)

Here is the code of the main function.

3. Effect display

This picture is the effect of recognition, and the frame rate is still very good.

3. Effect optimization

Although this frame rate is acceptable, the effect is a bit stretched. The senior who taught me K210 guided me to optimize and improve, here is the link of the senior (https://blog.csdn.net/hyayq8124?spm=1001.2014.3001.5509)

1. Specific code

import cv2
import time
import mediapipe as mp
from tqdm import tqdm

# 导入solution
mp_pose = mp.solutions.pose


mp_drawing = mp.solutions.drawing_utils


pose = mp_pose.Pose(static_image_mode=False,
            #    model_complexity=1,
                    smooth_landmarks=True,
           #        enable_segmentation=True,
                    min_detection_confidence=0.5,
                    min_tracking_confidence=0.5)


def process_frame(img):
    # BGR转RGB
    img_RGB = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)

    results = pose.process(img_RGB)

    # 可视化
    mp_drawing.draw_landmarks(img, results.pose_landmarks, mp_pose.POSE_CONNECTIONS)
    # look_img(img)

    # mp_drawing.plot_landmarks(results.pose_world_landmarks, mp_pose.POSE_CONNECTIONS)

    #     # BGR转RGB
    #     img_RGB = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)
    #
    #     results = hands.process(img_RGB)

    #     if results.multi_hand_landmarks: # 如果有检测到手
    #
    #         for hand_idx in range(len(results.multi_hand_landmarks)):
    #             hand_21 = results.multi_hand_landmarks[hand_idx]
    #             mpDraw.draw_landmarks(img, hand_21, mp_hands.HAND_CONNECTIONS)

    return img

cap = cv2.VideoCapture(1)

# 打开cap
cap.open(0)

# 无限循环，直到break被触发
while cap.isOpened():
    # 获取画面
    success, frame = cap.read()
    if not success:
        print('Error')
        break

    ## !!!处理帧函数
    frame = process_frame(frame)

    # 展示处理后的三通道图像
    cv2.imshow('my_window', frame)

    if cv2.waitKey(1) in [ord('q'), 27]:
        break


cap.release()


cv2.destroyAllWindows()

2. Effect display

The effect is simply too good. Hahaha

Summarize

This article is over here. Writing this blog is just to record my own learning process. I hope that you who read this blog can learn more firmly. Hu Shi said something that I think is particularly good, and I would like to share it with you here.
I am afraid that the truth is endless, and every inch of progress brings joy to every inch of progress.