As a competition in the list of the National College Students Competition, the "China Software Cup" College Students Software Design Competition - Intelligent Quadruped Robot Dog Power Inspection System Development Competition has attracted 2041 teams from all over the country to participate. After intense target detection and segmentation algorithm competition, 153 teams successfully advanced to the regional competition. In the regional competition to be held in mid-July, the qualified players must perform automatic inspections, obstacle avoidance and completion of designated tasks on the specified map based on the domestic quadruped robot dog Yushu Go1 robot dog equipped with a flying paddle frame , earn points based on the quality and speed of completion of each task.
Game simulation map
Developer Introduction #
Anhong County
Dalian Maritime University is a third-year student majoring in Intelligent Science and Technology, and his research direction is machine vision and intelligent autonomous robot. The national championship of the underwater operation group of the RoboCup Robot World Cup China Competition; the national championship of the material delivery track of the RoboCom Robot Developer Contest for College Students; the national championship of the Flying Paddle Track of the Chinese College Student Computer Design Competition; the National College Student Embedded Chip and System Design Competition Hisilicon National Championship. Recommended to be sent to Northwestern Polytechnical University Institute of Optoelectronics and Intelligence to study for doctoral students.
Mo Shanbin
Dalian Maritime University, majoring in electronic information science and technology, focusing on machine vision and hardware machine control. Extensively involved in various competitions, the 2022 ROBCOM material delivery champion team player, familiar with the control logic of the quadruped robot dog and its underlying motion control, has certain development experience; in the National Undergraduate Electronic Design Competition, Blue Bridge Cup and other embedded hardware development Repeated successes in competitions.
Expected goal: see the effect first
When the like gesture is detected, follow the movement
When the 666 gesture is detected, the puppy dances
When the STOP gesture is detected, it will stay still and follow
The Road to Domestication 1: Partial Realization of Perception
Perception part of the work goal: use the mini-HaGRID dataset and use the PaddleDetection tool to train a target detection model that can recognize three gestures like, call, and stop.
Server-side model training
This time, a simplified version of the HaGRID gesture detection dataset: mini-HaGRID is used as the training data. Of course, you can also use the Labelme labeling tool to make your own datasets, or find public datasets on the Internet. Use Labelme to label the dataset, you can refer to the tutorial: https://gitee.com/paddlepaddle/PaddleDetection/blob/release/2.6/docs/tutorials/data/DetAnnoTools.md
On AI Studio, train the deep learning vision model based on the data.
1.1 Data preparation
# 解压数据集
!unzip data/data215595/mini-HaGRID.zip -d data/mini-HaGRID
1.2 Download and install PaddleDetection v2.4
# 下载PaddleDetection仓库
!git clone https://gitee.com/paddlepaddle/PaddleDetection.git
# 更新pip版本
!pip install --upgrade pip
# 安装其他依赖
%cd PaddleDetection
!pip install -r requirements.txt --user
# 编译安装paddledet
!python setup.py install
%cd ~
1.3 Create a new dataset configuration file
Enter the PaddleDetection/configs/datasets directory, create a new file voc_hand.yml and use voc.yml in the same directory as a template, and modify the root directory of the dataset, the number of dataset categories and the data list file. The following content can be copied directly:
metric: VOC
map_type: 11point
num_classes: 4
TrainDataset:
!VOCDataSet
dataset_dir: /home/aistudio/data/mini-HaGRID
anno_path: train.txt
label_list: label_list.txt
data_fields: ['image', 'gt_bbox', 'gt_class', 'difficult']
EvalDataset:
!VOCDataSet
dataset_dir: /home/aistudio/data/mini-HaGRID
anno_path: val.txt
label_list: label_list.txt
data_fields: ['image', 'gt_bbox', 'gt_class', 'difficult']
TestDataset:
!ImageFolder
anno_path: /home/aistudio/data/mini-HaGRID/label_list.txt
1.4 Create a new training configuration file
Enter the PaddleDetection/configs/picodet directory, create a new configuration file picodet_xs_320_voc_hand.yml, use picodet_xs_320_coco_lcnet.yml in the same directory as a template, and modify the data set configuration. The following content can be copied directly:
_BASE_: [
'../datasets/voc_hand.yml',
'../runtime.yml',
'_base_/picodet_v2.yml',
'_base_/optimizer_300e.yml',
'_base_/picodet_320_reader.yml',
]
pretrain_weights: https://paddledet.bj.bcebos.com/models/pretrained/LCNet_x0_35_pretrained.pdparams
weights: output/picodet_xs_320_voc_hand/best_model
find_unused_parameters: True
use_ema: true
epoch: 300
snapshot_epoch: 10
LCNet:
scale: 0.35
feature_maps: [3, 4, 5]
LCPAN:
out_channels: 96
PicoHeadV2:
conv_feat:
name: PicoFeat
feat_in: 96
feat_out: 96
num_convs: 2
num_fpn_stride: 4
norm_type: bn
share_cls_reg: True
use_se: True
feat_in_chan: 96
TrainReader:
batch_size: 64
LearningRate:
base_lr: 0.32
schedulers:
- !CosineDecay
max_epochs: 300
- !LinearWarmup
start_factor: 0.1
steps: 300
1.5 Training Model
# 安装VisualDL可视化工具
!python -m pip install visualdl -i https://mirror.baidu.com/pypi/simple
# 训练模型
%cd ~/PaddleDetection
!python -u tools/train.py -c configs/picodet/picodet_xs_320_voc_hand.yml --use_vdl=true --vdl_log_dir=vdl_dir/scalar --eval
训练时，可以用VisualDL工具可视化训练参数。
visualdl --logdir vdl_dir/scalar/ --host <host_IP> --port <port_num>
Model export
After training the model using PaddleDetection, the model needs to be exported.
2.1 Export to PaddleInference format
# 导出模型
%cd ~/PaddleDetection
!python tools/export_model.py \
-c configs/picodet/picodet_xs_320_voc_hand.yml \
--output_dir=./inference_model -o weights=output/picodet_xs_320_voc_hand/best_model.pdparams
2.2 Install PaddleLite on the server side
Open the official github repository of PaddleLite: https://github.com/PaddlePaddle/Paddle-Lite Open the Release page, download the v2.12 version of opt_linux to the root directory of PaddleDetection. You can directly execute the following command:
%cd ~/PaddleDetection
!wget https://github.com/PaddlePaddle/Paddle-Lite/releases/download/v2.12/opt_linux
# 为opt_linux工具添加运行权限
%cd ~/PaddleDetection
!chmod +x opt_linux
2.3 Export the model to PaddleLite format
%cd ~/PaddleDetection
!mkdir inference_model_opt
!./opt_linux --model_dir=./inference_model/picodet_xs_320_voc_hand --valid_targets=arm --optimize_out_type=naive_buffer --optimize_out=./inference_model_opt/voc_hand
Among them, the more reliable instructions of the opt_linux tool can refer to https://www.paddlepaddle.org.cn/lite/v2.12/user_guides/opt/opt_bin.html After running, you can see a file under the inference_model_opt folder "voc_hand.nb", this is the converted PaddleLite model file.
visual model
3.1 Download model file
After the training is completed, download a file "voc_hand.nb" under the inference_model_opt folder. Open the model visualization website: https://netron.app/ and upload the voc_hand.nb model.
3.2 Record input and output names
The model visualization is shown in the figure: 
click the input node "0", and the sidebar will pop up: 
you can see that the model format is PaddleLite, and the model has two input nodes and two output nodes.
The meaning of each node can refer to: https://gitee.com/paddlepaddle/PaddleDetection/blob/release/2.6/deploy/EXPORT_MODEL.md
- scale_factor: scaling factor
- image: input image
- multiclass_nms3_0.tmp_0: Candidate results after NMS
- multiclass_nms3_0.tmp_2: the number of candidate results
Remember the names of these four nodes , they will be used during deployment.
Road to Domestication II: Partial Implementation of Hardware
Next, we will teach how to deploy the trained voc_hand model to Yushu Go1 quadruped robot, and use gestures to control the robot dog to complete the response action.
Yushu Go1 robot dog is built with 3 Nano and 1 Raspberry Pi
- Clone this project to the home directory of Nano1 and Raspberry Pi.
cd ~
git clone https://gitee.com/an_hongjun/2023_paddle_dog_demo.git
cd 2023_paddle_dog_demo
Nano1 Deployment
Enter the nano1-workspace directory
cd nano1-workspace
kill process
ps -aux | grep point_cloud_node | awk '{print $2}' | xargs kill -9
ps -aux | grep mqttControlNode | awk '{print $2}' | xargs kill -9
ps -aux | grep live_human_pose | awk '{print $2}' | xargs kill -9
ps -aux | grep rosnode | awk '{print $2}' | xargs kill -9
modify system time
Change the time to your current time. The wrong time may affect the compilation of the CMake program.
sudo date -s "2023-6-11 21:23:00"
compile program
./build.sh
Run the nano1 node
./bin/det_hand
Raspberry Pi Deployment
Enter the pi-workspace directory
cd pi-workspace
modify system time
Change the time to your current time. Wrong time may affect CMake program compilation.
sudo date -s "2023-6-11 21:23:00"
compile program
./build.sh
run pi-node
./bin/follow
code explanation
Nano1 node code explained
https://gitee.com/an_hongjun/2023_paddle_dog_demo/blob/master/nano1-workspace/README.md
Pi node code explained
https://gitee.com/an_hongjun/2023_paddle_dog_demo/blob/master/pi-workspace/README.md
Conclusion: Developers have something to say
As an author, I must admit that the solution proposed in this project is not necessarily the best. for example:
- There is still a lot of room for model selection;
- When the model is exported, the compression of the model can also do a lot of articles (maybe PaddleSlim is a good reference);
- When the model is deployed, the code still has a lot of room for optimization, and it may be possible to perform faster reasoning in combination with TensorRT.
The purpose of this project is mainly to help everyone get started. Maybe it's not the best, but a good start. I wish you all the best for the game!