【基于PaddleHub的跳绳AI计数器】智能计数器

我正在参加「创意开发 投稿大赛」详情请看：掘金创意开发大赛来了！

一、【基于PaddleHub的跳绳AI计数器】智能计数介绍

1.背景介绍

疫情以来，小朋友、大朋友们非必要不外出，都老老实实的画地为牢了。跳绳这类比起跑步等户外活动的优势一下子凸显了出来。值此佳节【六一儿童节】到来之际，我给大家弄一个通过手机视频计数的APP方案，具体如下，我们小学生也会做啦：

2.实现思路

• 1.用户打开手机APP，将手机固定在场地一侧，适当设置手机角度，根据应用的自动语音提示调整身体与手机距离，直到人体完全位于识别框内，即可开始运动。
• 2.通过PaddleHub的human_pose_estimation_resnet50_mpii模型，进行人体关键点检测。
• 3.根据检测的数据，计算是否起跳，并计数（此处选择骨盆关键点进行判断，因为跳绳有多种花式，选择其他关键点不一定能记录到），
• 4.移植该应用到手机app端

另外，【六一】活动大家可以积极相应，具体地址：

精品项目征集方向（一）-青少年主题项目征集 aistudio.baidu.com/paddle/foru…

二、环境准备

1.PaddleHub安装

!pip install -U pip --user >log.log
!pip install -U paddlehub >log.log

ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
parl 1.4.1 requires pyzmq==18.1.1, but you have pyzmq 22.3.0 which is incompatible.

!pip list |grep paddle

paddle2onnx            0.9.5
paddlehub              2.2.0
paddlenlp              2.0.1
paddlepaddle           2.2.2
tb-paddle              0.3.6

2.human_pose_estimation_resnet50_mpii模型安装

• 模型地址： www.paddlepaddle.org.cn/hubdetail?n…
• 模型概述：人体骨骼关键点检测(Pose Estimation) 是计算机视觉的基础性算法之一，在诸多计算机视觉任务起到了基础性的作用，如行为识别、人物跟踪、步态识别等相关领域。具体应用主要集中在智能视频监控，病人监护系统，人机交互，虚拟现实，人体动画，智能家居，智能安防，运动员辅助训练等等。 该模型的论文《Simple Baselines for Human Pose Estimation and Tracking》由 MSRA 发表于 ECCV18，使用 MPII 数据集训练完成。

!hub install human_pose_estimation_resnet50_mpii >log.log

/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/setuptools/depends.py:2: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
  import imp
/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/scipy/sparse/sputils.py:16: DeprecationWarning: `np.typeDict` is a deprecated alias for `np.sctypeDict`.
  supported_dtypes = [np.typeDict[x] for x in supported_dtypes]
/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/scipy/special/orthogonal.py:81: DeprecationWarning: `np.int` is a deprecated alias for the builtin `int`. To silence this warning, use `int` by itself. Doing this will not modify any behavior and is safe. When replacing `np.int`, you may wish to use e.g. `np.int64` or `np.int32` to specify the precision. If you wish to review your current use, check the release note link for additional information.
Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations
  from numpy import (exp, inf, pi, sqrt, floor, sin, cos, around, int,
/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/scipy/linalg/__init__.py:217: DeprecationWarning: The module numpy.dual is deprecated.  Instead of using dual, use the functions directly from numpy or scipy.
  from numpy.dual import register_func
/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/onnx/mapping.py:27: DeprecationWarning: `np.object` is a deprecated alias for the builtin `object`. To silence this warning, use `object` by itself. Doing this will not modify any behavior and is safe. 
Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations
  int(TensorProto.STRING): np.dtype(np.object)
[2022-05-03 23:56:11,218] [    INFO] - Successfully installed human_pose_estimation_resnet50_mpii-1.1.1

!hub list|grep human

/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/setuptools/depends.py:2: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
  import imp
/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/scipy/sparse/sputils.py:16: DeprecationWarning: `np.typeDict` is a deprecated alias for `np.sctypeDict`.
  supported_dtypes = [np.typeDict[x] for x in supported_dtypes]
/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/scipy/special/orthogonal.py:81: DeprecationWarning: `np.int` is a deprecated alias for the builtin `int`. To silence this warning, use `int` by itself. Doing this will not modify any behavior and is safe. When replacing `np.int`, you may wish to use e.g. `np.int64` or `np.int32` to specify the precision. If you wish to review your current use, check the release note link for additional information.
Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations
  from numpy import (exp, inf, pi, sqrt, floor, sin, cos, around, int,
/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/scipy/linalg/__init__.py:217: DeprecationWarning: The module numpy.dual is deprecated.  Instead of using dual, use the functions directly from numpy or scipy.
  from numpy.dual import register_func
/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/onnx/mapping.py:27: DeprecationWarning: `np.object` is a deprecated alias for the builtin `object`. To silence this warning, use `object` by itself. Doing this will not modify any behavior and is safe. 
Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations
  int(TensorProto.STRING): np.dtype(np.object)
|human_pose_estimation_res|/home/aistudio/.paddlehub/modules/human_pose_estim|

三、人体关键点检测示例

1.抓取视频

可以通过数据集中的小助手工具，下载抖音、西瓜视频、快手等小视频，特别是可以下载所需要的跳绳视频，具体演示如下：

2.关键点检测演示

针对下面这张图片做关键点检测，具体如下：

import cv2
import paddlehub as hub

pose_estimation = hub.Module(name="human_pose_estimation_resnet50_mpii")
image=cv2.imread('data/data143990/1.jpg')
results = pose_estimation.keypoint_detection(images=[image], visualization=True)

[2022-05-03 23:56:14,417] [ WARNING] - The _initialize method in HubModule will soon be deprecated, you can use the __init__() to handle the initialization of the object
W0503 23:56:14.420603    97 analysis_predictor.cc:1350] Deprecated. Please use CreatePredictor instead.


image saved in output_pose/ndarray_time=1651593374766039.jpg

# 打印关键点
print(results[0]['data'])

OrderedDict([('left_ankle', [104, 376]), ('left_knee', [107, 312]), ('left_hip', [117, 243]), ('right_hip', [142, 243]), ('right_knee', [148, 312]), ('right_ankle', [163, 381]), ('pelvis', [132, 243]), ('thorax', [132, 188]), ('upper_neck', [132, 179]), ('head_top', [130, 133]), ('right_wrist', [84, 252]), ('right_elbow', [102, 225]), ('right_shoulder', [112, 188]), ('left_shoulder', [153, 183]), ('left_elbow', [160, 225]), ('left_wrist', [183, 252])])

查看output_pose 下输出的图片：

3.寻找转折点

众所周知，跳绳，人体每一帧盆骨点变化的时候算一次，那么可以通过下面方法来计算次数，具体如下：

# -*- coding: utf-8 -*-
# __author__:Livingbody
# 2022/5/2 21:26

import math
from matplotlib import pyplot as plt
import numpy as np
%matplotlib inline


def get_count(y):
    count = 0
    flag = False
    for i in range(len(y)-1):
        if y[i] < y[i + 1] and flag == False:
            continue
        elif y[i] > y[i + 1] and flag == True:
            continue
        else:
            print(i, y[i])
            count = count + 1
            flag = not flag
    return count

def get_count(y):
    count = 0
    flag = False
    for i in range(len(y)-1):
        if y[i] < y[i + 1] and flag == False:
            continue
        elif y[i] > y[i + 1] and flag == True:
            continue
        else:
            print(i, y[i])
            count = count + 1
            flag = not flag
    return count


if __name__ == '__main__':
    x = np.linspace(-10, 10, 500)
    y = np.sin(x * math.pi)
    # print(x, y)
    plt.figure(figsize=(12, 12))
    count = get_count(y)
    plt.title(f"point numbers: {math.floor(count/2)}")
    plt.plot(x, y)
    plt.show()

12 0.9982119222410882
37 -0.9985684620536466
62 0.9988854219322505
87 -0.999162789313664
112 0.9994005532039664
137 -0.9995987041789873
162 0.999757234384681
187 -0.9998761375374373
212 0.9999554089243304
237 -0.9999950454033061
262 0.9999950454033061
287 -0.9999554089243304
312 0.9998761375374373
337 -0.9997572343846811
362 0.9995987041789874
387 -0.9994005532039665
412 0.999162789313664
437 -0.9988854219322507
462 0.9985684620536466
487 -0.9982119222410885


/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/matplotlib/cbook/__init__.py:2349: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working
  if isinstance(obj, collections.Iterator):
/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/matplotlib/cbook/__init__.py:2366: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working
  return list(data) if isinstance(data, collections.MappingView) else data

如上图所示，可以算出该曲线里有10次最高点，该方法可运用到跳绳计数中去。

四、跳绳检测

# -*- coding: utf-8 -*-
# __author__:Livingbody
# 2022/5/2 17:49


import cv2
import paddlehub as hub

if __name__ == "__main__":
    pose_estimation = hub.Module(name="human_pose_estimation_resnet50_mpii")

    flag = False
    count = 0
    flip_list = []
    # 可选择web视频流或者文件
    file_name = 'data/data143990/2.mp4'
    cap = cv2.VideoCapture(file_name)
    fourcc = cv2.VideoWriter_fourcc(*"mp4v")
    # out后期可以合成视频返回
    out = cv2.VideoWriter(
        file_name.replace(".mp4", "_output.mp4"),
        fourcc,
        20.0,
        (int(cap.get(3)), int(cap.get(4))), )

    while cap.isOpened():
        success, image = cap.read()
        if not success:
            break
        image_height, image_width, _ = image.shape

        image.flags.writeable = False
        results = pose_estimation.keypoint_detection(images=[image], visualization=True)
        # debug打印，可关闭
        # print(results)
        # print(results[0]['data'])
        # print(results[0]['data']['pelvis'])
        # print(results[0]['data']['left_hip'])
        # print(results[0]['data']['right_hip'])
        flip = results[0]['data']['pelvis'][1]
        flip_list.append(flip)
        # 如果超过2个点就开始比较
        if len(flip_list) < 2:
            continue
        prev_flip = flip_list[len(flip_list) - 2]
        
        # 开始进行判断计数
        if flip < prev_flip and flag == False:
            continue
        elif flip > prev_flip and flag == True:
            continue
        else:
            # print(flip)
            count = count + 1
            flag = not flag
        print(int(count/2))

输出示例：

image saved in output_pose/ndarray_time=1651593377511581.jpg
image saved in output_pose/ndarray_time=1651593378354247.jpg
0
image saved in output_pose/ndarray_time=1651593378987333.jpg
1
image saved in output_pose/ndarray_time=1651593379606857.jpg
1
image saved in output_pose/ndarray_time=1651593380213530.jpg
image saved in output_pose/ndarray_time=1651593380818746.jpg
image saved in output_pose/ndarray_time=1651593381474900.jpg
2
image saved in output_pose/ndarray_time=1651593382093679.jpg
2
image saved in output_pose/ndarray_time=1651593382712164.jpg
3
image saved in output_pose/ndarray_time=1651593383325455.jpg
3

五、下步打算

该项目抛砖引玉，供大家参考。

• 加入可视化，实时显示关键点检测以及跳绳计数结果；
• 拟采取paddlelite终端部署，或者paddlehub服务器部署调用，结合android客户端、或者电脑加摄像头实现实现方式来验证；
• 数据下载软件已上传数据集，大家也可以动手试试。