零、写在前面
由于距离写这篇文章过去很久，有的部分，官方已更新，请多结合其他人的看，并多琢磨、讨论~
另外打个小广告：博客 https://blog.vrxiaojie.top/ 欢迎大家前来做客玩耍，提出问题~~
以后的文章都会在博客发布了，CSDN这边可能这是最后一篇文章。
一、烧录系统
1.准备工作：
(1) 使用官方提供的Ubuntu镜像：
Orange Pi 5系统下载（我使用Orangepi5_1.1.4_ubuntu_jammy_desktop_gnome_linux5.10.110）
(2) 下载 balenaEtcher 用于烧录系统
(3) 准备一个大于等于32GB的TF卡、一个读卡器。
2.打开软件，根据软件步骤烧录即可
完成后，插上TF卡、键盘、鼠标，上电开机。
二、在开发板上的准备工作
1. 在终端查看ip，并用ssh连接
香橙派接显示器，与电脑连接同一个网络，
使用 ip address查看ip
使用任意一个ssh软件，连接
默认的用户名有: root、orangepi，它们的默认密码都是orangepi
2. 安装ftp服务器，并连接
安装
sudo apt-get install vsftpd
修改配置文件
sudo vim /etc/vsftpd.conf
local_enable=YES
write_enable=YES
并确认它们前面没有井号(vim操作：按i键进入编辑模式，esc后输入:wq退出保存)
重启ftp服务
sudo /etc/init.d/vsftpd restart
filezilla设置
主机：香橙派的ip
用户名:orangepi
密码: orangepi
三、pc端的准备工作
1.准备一台装有Windows10及以上系统的电脑，并用WSL装Ubuntu22.04系统
WSl2安装过程参考微软官方教程
请自己手动配置好Ubuntu的运行环境以及conda，这里不多赘述。
2.下载rknn_toolkit2的所有源码 github页面 并解压
3.下载 yolov5官方源码，并训练模型
为节省时间，在这里先不自己训练了，就使用预训练权重yolov5n.pt
4.将pt转换成onnx格式
首先，根据官方文档的要求，在训练完成模型之后，再修改models/yolo.py Detect类下的forward函数
也就是修改为
def forward(self, x):
z = [] # inference output
for i in range(self.nl):
x[i] = self.m[i](x[i]) # conv
return x
然后，在yolov5目录下，打开终端输入命令
python export.py --weights yolov5n.pt --data data/coco128.yaml --include onnx --opset 12 --batch-size 1
注意注意！opset一定要为12，不然后面onnx转rknn会报错。weights自己选你训练完成的best.pt，data选你自己设置的，我这里只是做一个最简单情况的演示。
这里我遇到一个错误:
line 715, in run
shape = tuple((y[0] if isinstance(y, tuple) else y).shape) # model output shape
AttributeError: ‘list’ object has no attribute ‘shape’
解决方法
找到这一行，修改为
shape = tuple(y[0].shape) # model output shape
程序运行结束后，会在当前文件夹下生成一个yolov5n.onnx文件，待会要用。
5. onnx转化为rknn格式
进入你的Ubuntu系统，用conda创建一个新的环境
conda create -name rknn python=3.8
激活这个环境
conda activate rknn
将整个rknn_toolkit2源码解压后的文件夹复制进Ubuntu的 /home/用户名目录下，我将它重命名为rknn。
进入doc目录，使用 pip 安装依赖:
pip install -r requirements-cpu-ubuntu20.04_py38.txt -i https://mirror.baidu.com/pypi/simple
返回上一级目录，然后进入packages目录，安装rknn_toolkit2
pip install rknn_toolkit2-1.4.0_22dcfef4-cp38-cp38-linux_x86_64.whl
完成后，输入命令 python
from rknn.api import RKNN
如图所示，如果没有报错，说明安装成功，使用键盘Ctrl+Z退出此模式。
接下来，把(三、4)步所得到的onnx文件放入rknn/examples/onnx/yolov5文件夹下，终端里进入该文件夹。
用你喜欢的编辑器修改 test.py里面的一些内容，具体位置如图所示
-
onnx和要生成的rknn的文件名
-
classes改成你模型的类型，我这里是官方的所以默认
-
增加target_platform=‘rk3588’，根据你的板子型号自行调整
执行
python test.py
成功运行后的内容大概是这样的
此时这个目录下也会生成一个yolov5n.rknn文件，待会要用。
四、在香橙派上的部署
1.创建一个名为deploy.py的文件
内容如下
import numpy as np
import cv2
from rknnlite.api import RKNNLite
RKNN_MODEL = 'yolov5n.rknn'
QUANTIZE_ON = True
OBJ_THRESH = 0.25
NMS_THRESH = 0.45
IMG_SIZE = 640
CLASSES = ("person", "bicycle", "car", "motorbike ", "aeroplane ", "bus ", "train", "truck ", "boat", "traffic light",
"fire hydrant", "stop sign ", "parking meter", "bench", "bird", "cat", "dog ", "horse ", "sheep", "cow", "elephant",
"bear", "zebra ", "giraffe", "backpack", "umbrella", "handbag", "tie", "suitcase", "frisbee", "skis", "snowboard", "sports ball", "kite",
"baseball bat", "baseball glove", "skateboard", "surfboard", "tennis racket", "bottle", "wine glass", "cup", "fork", "knife ",
"spoon", "bowl", "banana", "apple", "sandwich", "orange", "broccoli", "carrot", "hot dog", "pizza ", "donut", "cake", "chair", "sofa",
"pottedplant", "bed", "diningtable", "toilet ", "tvmonitor", "laptop ", "mouse ", "remote ", "keyboard ", "cell phone", "microwave ",
"oven ", "toaster", "sink", "refrigerator ", "book", "clock", "vase", "scissors ", "teddy bear ", "hair drier", "toothbrush ")
def sigmoid(x):
return 1 / (1 + np.exp(-x))
def xywh2xyxy(x):
# Convert [x, y, w, h] to [x1, y1, x2, y2]
y = np.copy(x)
y[:, 0] = x[:, 0] - x[:, 2] / 2 # top left x
y[:, 1] = x[:, 1] - x[:, 3] / 2 # top left y
y[:, 2] = x[:, 0] + x[:, 2] / 2 # bottom right x
y[:, 3] = x[:, 1] + x[:, 3] / 2 # bottom right y
return y
def process(input, mask, anchors):
anchors = [anchors[i] for i in mask]
grid_h, grid_w = map(int, input.shape[0:2])
box_confidence = sigmoid(input[..., 4])
box_confidence = np.expand_dims(box_confidence, axis=-1)
box_class_probs = sigmoid(input[..., 5:])
box_xy = sigmoid(input[..., :2])*2 - 0.5
col = np.tile(np.arange(0, grid_w), grid_w).reshape(-1, grid_w)
row = np.tile(np.arange(0, grid_h).reshape(-1, 1), grid_h)
col = col.reshape(grid_h, grid_w, 1, 1).repeat(3, axis=-2)
row = row.reshape(grid_h, grid_w, 1, 1).repeat(3, axis=-2)
grid = np.concatenate((col, row), axis=-1)
box_xy += grid
box_xy *= int(IMG_SIZE/grid_h)
box_wh = pow(sigmoid(input[..., 2:4])*2, 2)
box_wh = box_wh * anchors
box = np.concatenate((box_xy, box_wh), axis=-1)
return box, box_confidence, box_class_probs
def filter_boxes(boxes, box_confidences, box_class_probs):
"""Filter boxes with box threshold. It's a bit different with origin yolov5 post process!
# Arguments
boxes: ndarray, boxes of objects.
box_confidences: ndarray, confidences of objects.
box_class_probs: ndarray, class_probs of objects.
# Returns
boxes: ndarray, filtered boxes.
classes: ndarray, classes for boxes.
scores: ndarray, scores for boxes.
"""
boxes = boxes.reshape(-1, 4)
box_confidences = box_confidences.reshape(-1)
box_class_probs = box_class_probs.reshape(-1, box_class_probs.shape[-1])
_box_pos = np.where(box_confidences >= OBJ_THRESH)
boxes = boxes[_box_pos]
box_confidences = box_confidences[_box_pos]
box_class_probs = box_class_probs[_box_pos]
class_max_score = np.max(box_class_probs, axis=-1)
classes = np.argmax(box_class_probs, axis=-1)
_class_pos = np.where(class_max_score >= OBJ_THRESH)
boxes = boxes[_class_pos]
classes = classes[_class_pos]
scores = (class_max_score* box_confidences)[_class_pos]
return boxes, classes, scores
def nms_boxes(boxes, scores):
"""Suppress non-maximal boxes.
# Arguments
boxes: ndarray, boxes of objects.
scores: ndarray, scores of objects.
# Returns
keep: ndarray, index of effective boxes.
"""
x = boxes[:, 0]
y = boxes[:, 1]
w = boxes[:, 2] - boxes[:, 0]
h = boxes[:, 3] - boxes[:, 1]
areas = w * h
order = scores.argsort()[::-1]
keep = []
while order.size > 0:
i = order[0]
keep.append(i)
xx1 = np.maximum(x[i], x[order[1:]])
yy1 = np.maximum(y[i], y[order[1:]])
xx2 = np.minimum(x[i] + w[i], x[order[1:]] + w[order[1:]])
yy2 = np.minimum(y[i] + h[i], y[order[1:]] + h[order[1:]])
w1 = np.maximum(0.0, xx2 - xx1 + 0.00001)
h1 = np.maximum(0.0, yy2 - yy1 + 0.00001)
inter = w1 * h1
ovr = inter / (areas[i] + areas[order[1:]] - inter)
inds = np.where(ovr <= NMS_THRESH)[0]
order = order[inds + 1]
keep = np.array(keep)
return keep
def yolov5_post_process(input_data):
masks = [[0, 1, 2], [3, 4, 5], [6, 7, 8]]
anchors = [[10, 13], [16, 30], [33, 23], [30, 61], [62, 45],
[59, 119], [116, 90], [156, 198], [373, 326]]
boxes, classes, scores = [], [], []
for input, mask in zip(input_data, masks):
b, c, s = process(input, mask, anchors)
b, c, s = filter_boxes(b, c, s)
boxes.append(b)
classes.append(c)
scores.append(s)
boxes = np.concatenate(boxes)
boxes = xywh2xyxy(boxes)
classes = np.concatenate(classes)
scores = np.concatenate(scores)
nboxes, nclasses, nscores = [], [], []
for c in set(classes):
inds = np.where(classes == c)
b = boxes[inds]
c = classes[inds]
s = scores[inds]
keep = nms_boxes(b, s)
nboxes.append(b[keep])
nclasses.append(c[keep])
nscores.append(s[keep])
if not nclasses and not nscores:
return None, None, None
boxes = np.concatenate(nboxes)
classes = np.concatenate(nclasses)
scores = np.concatenate(nscores)
return boxes, classes, scores
def draw(image, boxes, scores, classes):
"""Draw the boxes on the image.
# Argument:
image: original image.
boxes: ndarray, boxes of objects.
classes: ndarray, classes of objects.
scores: ndarray, scores of objects.
all_classes: all classes name.
"""
for box, score, cl in zip(boxes, scores, classes):
top, left, right, bottom = box
print('class: {}, score: {}'.format(CLASSES[cl], score))
print('box coordinate left,top,right,down: [{}, {}, {}, {}]'.format(top, left, right, bottom))
top = int(top)
left = int(left)
right = int(right)
bottom = int(bottom)
cv2.rectangle(image, (top, left), (right, bottom), (255, 0, 0), 2)
cv2.putText(image, '{0} {1:.2f}'.format(CLASSES[cl], score),
(top, left - 6),
cv2.FONT_HERSHEY_SIMPLEX,
0.6, (0, 0, 255), 2)
def letterbox(im, new_shape=(640, 640), color=(0, 0, 0)):
# Resize and pad image while meeting stride-multiple constraints
shape = im.shape[:2] # current shape [height, width]
if isinstance(new_shape, int):
new_shape = (new_shape, new_shape)
# Scale ratio (new / old)
r = min(new_shape[0] / shape[0], new_shape[1] / shape[1])
# Compute padding
ratio = r, r # width, height ratios
new_unpad = int(round(shape[1] * r)), int(round(shape[0] * r))
dw, dh = new_shape[1] - new_unpad[0], new_shape[0] - new_unpad[1] # wh padding
dw /= 2 # divide padding into 2 sides
dh /= 2
if shape[::-1] != new_unpad: # resize
im = cv2.resize(im, new_unpad, interpolation=cv2.INTER_LINEAR)
top, bottom = int(round(dh - 0.1)), int(round(dh + 0.1))
left, right = int(round(dw - 0.1)), int(round(dw + 0.1))
im = cv2.copyMakeBorder(im, top, bottom, left, right, cv2.BORDER_CONSTANT, value=color) # add border
return im, ratio, (dw, dh)
if __name__ == '__main__':
# Create RKNN object
rknn = RKNNLite()
# load RKNN model
print('--> Load RKNN model')
ret = rknn.load_rknn(RKNN_MODEL)
# Init runtime environment
print('--> Init runtime environment')
ret = rknn.init_runtime(core_mask=RKNNLite.NPU_CORE_0_1_2) #使用0 1 2三个NPU核心
# ret = rknn.init_runtime('rk3566')
if ret != 0:
print('Init runtime environment failed!')
exit(ret)
print('done')
# Set inputs
img = cv2.imread(IMG_PATH)
# img, ratio, (dw, dh) = letterbox(img, new_shape=(IMG_SIZE, IMG_SIZE))
img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)
img = cv2.resize(img, (IMG_SIZE, IMG_SIZE))
# Inference
outputs = rknn.inference(inputs=[img])
# post process
input0_data = outputs[0]
input1_data = outputs[1]
input2_data = outputs[2]
input0_data = input0_data.reshape([3, -1]+list(input0_data.shape[-2:]))
input1_data = input1_data.reshape([3, -1]+list(input1_data.shape[-2:]))
input2_data = input2_data.reshape([3, -1]+list(input2_data.shape[-2:]))
input_data = list()
input_data.append(np.transpose(input0_data, (2, 3, 0, 1)))
input_data.append(np.transpose(input1_data, (2, 3, 0, 1)))
input_data.append(np.transpose(input2_data, (2, 3, 0, 1)))
boxes, classes, scores = yolov5_post_process(input_data)
img_1 = cv2.cvtColor(img, cv2.COLOR_RGB2BGR)
if boxes is not None:
draw(img_1, boxes, scores, classes)
# show output
cv2.imshow("post process result", img_1)
cv2.waitKey(0)
cv2.destroyAllWindows()
rknn.release()
这是基于原来test.py做的一些修改，删除了导出模型的部分，只保留了加载模型和推理部分。
接下来把这几个文件放在同一个文件夹，传到香橙派的 /home/orangepi/Desktop 目录下
2.在香橙派上安装miniconda并创建环境
建议去清华源下载 miniconda
我下载的是Miniconda3-py39_4.9.2-Linux-aarch64.sh
下载完后传到香橙派，在终端输入
sudo bash Miniconda3-py39_4.9.2-Linux-aarch64.sh
根据提示安装即可。
安装完后，可能会遇到找不到conda命令的问题，解决方法参考：
https://blog.csdn.net/weixin_42314494/article/details/114264103
先换清华源
conda config --add channels https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/free/
conda config --add channels https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main/
conda config --set show_channel_urls yes
然后创建环境
conda create --name rknn python=3.9
激活环境
conda activate rknn
pip换清华源
pip config set global.index-url https://pypi.tuna.tsinghua.edu.cn/simple
把你PC上的rknn_toolkit2/rknn_toolkit_lite2/packages中的rknn_toolkit_lite2-1.4.0-cp39-cp39-linux_aarch64.whl传输到香橙派上
安装上步的包
pip install rknn_toolkit_lite2-1.4.0-cp39-cp39-linux_aarch64.whl
安装剩下的依赖包
pip install opencv-python
然后就是运行deploy.py
python deploy.py
在ssh终端运行的话，会显示如下界面，可以看到蓝色框内是表示识别出来的类别、置信度以及检测框的四个顶点位置
如果有外接显示器，你就能看到它处理后的图像了：
五、后记
基础的部署到这里就结束了，我在社区里看到几篇非常不错的文章，这里附上作者的链接，可供参考。
1.如果你要做视频检测，又发现帧数提不上来，请使用多线程：
2.模型转换时出现问题，可参考：
3.将视频推理完的结果进行推流：
RK3588实战：调用npu加速，yolov5识别图像、ffmpeg发送到rtmp服务器