Smart cities aim to use digital technologies such as big data, the Internet of Things (IoT), artificial intelligence, and 5G to improve the level of government public services, the effectiveness of social governance, promote economic growth, and continuously enhance the people's sense of gain, security, and happiness. Since the 14th Five-Year Plan, the country and major cities have been accelerating the construction of a new type of smart city classification and classification, but some problems and difficulties have also been encountered during the implementation process.
In the implementation of AI deployment in smart cities, common problems include inconsistency of edge device hardware, high cross-platform development costs, and difficulty in optimizing model inference speed . Focusing on the users of the AI system, there are also many obstacles to the implementation of smart cities. For example, when environmental engineers collect data outdoors, if they want to collect data in real time, analyze it and store it in storage, they must carry a workstation equipped with a graphics card to go outdoors . At the same time, it is necessary to bring a XXXXL mobile power supply to solve the power supply problem. Its weight is undoubtedly overloading for environmental engineers.
In order to solve the problem of difficult development for software engineers, this solution demonstrates the AI workflow mode of FastDeploy (OpenVINO) + lightweight model , hoping to provide users with a lighter, simpler and more efficient solution.
This project uses the real-time message transmission protocol (RTMP) that comes with the UAV control terminal to transmit the real-time image of the low-altitude UAV to the inference hardware device, and calculate the green coverage rate, building rate, number of vehicles, number of people and other indicators through model inference , can be used in large-scale urban environmental monitoring, low-altitude remote sensing of land, road traffic inspection, and low-altitude security of drones.
This task is real-time inference on UAV image data flow, which has extremely high requirements on inference speed. At the same time, in order to reduce the development difficulty and migration cost, we adopted the Intel NUC mini computer kit with X86 CPU architecture as the inference hardware, and the FastDeploy inference deployment toolbox was selected for the software to quickly develop the back-end OpenVINO inference engine to accelerate AI model inference. In addition, in the selection of AI models, we selected two lightweight models, Flying Paddle PP-LiteSeg and Flying Paddle PP-YOLO Tiny, to complete semantic segmentation and target detection tasks.
Overall project process
As shown in the figure above, firstly, the RTMP streaming service needs to be set up on the NUC, and the UAV APP client is connected to the RTMP server on the intranet, so that the images of the UAV can be transmitted to the NUC device through the router in real time. After the NUC device obtains the image, it renders it to the display window on the front end of PySide, and at the same time, the FastDeploy thread performs the reasoning tasks of semantic segmentation and object detection, and renders the result and the visualized image to the front window in real time.
Development environment preparation
Core hardware and software
Intel NUC Mini PC Kit
(Model: NUC8i5BEH)
FastDeploy >= 0.2.1
PaddleDetection >= 2.5
PaddleSeg >= 2.6
PySide6 >= 6.3.2
Optional software and hardware (for outdoor real-time acquisition)
DJI series drones and DJI Fly APP
Router (for intranet RTMP transmission)
Intel NUC Mini Computer Hardware
UAV real-time image transmission
RTMP (Real-Time Messaging Protocol, Real-Time Messaging Protocol) is a network protocol designed for real-time data communication. It is now mostly used for audio, video and data communication between live broadcast devices and servers supporting the RTMP protocol. The real-time image transmission link between the UAV and the inference hardware device in this project uses the RTMP streaming service of the DJI control terminal APP. In order to minimize the image transmission time delay, we directly use Nginx to build RTMP on the Intel NUC Serve.
The pre-installed operating system of Intel NUC is Windows 10, so you can directly download Nginx with RTMP module.
- download link
http://nginx-win.ecsds.eu/download/nginx%201.7.11.3%20Gryphon.zip
After downloading to the local, decompress it, create a new file in the nginx/conf directory, name it nginx.conf, and enter the following content.
worker_processes 1;events { worker_connections 1024;}rtmp { server { listen 8899; chunk_size 4000; application live { live on; allow publish all; allow play all; } }}
Then open the cmd command line window, enter the Nginx directory, and enter nginx to start the Nginx service. As shown in the figure, the startup is successful.
At this point, we need to enter the ipconfig command on the cmd command line to query the intranet IPv4 address of the NUC device.
Open the DJI Fly APP, find the RTMP live streaming, and fill in the streaming address: rtmp://192.168.31.246:8899/live
After the APP streaming configuration is completed, you can call the OpenCV API on the NUC device side to perform the streaming operation to obtain the real-time picture of the drone. The Python implementation code is as follows.
import cv2rtmpUrl = 'rtmp://192.168.31.246:8899/live'vid = cv2.VideoCapture(rtmpUrl)while vid.isOpened(): ret,frame = vid.read() cv2.imshow('RTMP Test',frame) if cv2.waitKey(1) & 0xFF == ord('q'): breakvid.release()cv2.destroyAllWindows()
After running this code, the Python program will obtain the real-time RTMP stream data pushed by the UAV APP in real time and update it to the OpenCV window for display. The running effect is as shown in the figure below.
FastDeploy
Model inference deployment
After getting through the link of real-time transmission of UAV images, we entered the core reasoning and deployment link. As can be seen from the above Python code, the RTMP data stream is decoded by OpenCV to obtain an image frame (frame), so the essence of the work of the reasoning link is to input each frame of image into the model and then obtain the result and visualize the output result. This link is mainly divided into three steps: dynamic and static model conversion, reasoning script writing and front-end integration.
Model dynamic and static conversion
First, we need to convert the dynamic graph model trained using the PaddleSeg and PaddleDetection development kits into a static graph model. This step can be easily completed using the scripts provided by the two kits. The trained dynamic graph model can be obtained in the AI Studio project.
- Link
https://aistudio.baidu.com/aistudio/projectdetail/4535658
The semantic segmentation object detection dynamic graph model and semantic segmentation are respectively in the ppyolo_inference_model and output_ppliteseg_stdc1/best_model/ directories in the AI Studio project, and the corresponding dynamic and static conversion scripts are called to convert the two dynamic graph models into static graphs.
- PaddleSeg
python PaddleSeg/export.py \ --config ppliteseg_stdc1.yml \ #配置文件路径，根据实际情况修改 --model_path output_ppliteseg_stdc1/best_model/model.pdparams \ # 动态图模型路径，根据实际情况修改 --save_dir inference_ppliteseg_stdc1 \ # 导出目录 --input_shape 1 3 512 1024 #模型输入尺寸
- PaddleDetection
python PaddleDetection/tools/export_model.py \ -c PaddleDetection/configs/ppyolo/ppyolo_tiny_650e_coco.yml \ # 配置文件路径，根据实际情况修改 --output_dir=./ppyolo_inference_model \ # 保存模型的路径 -o weights=ppyolo_tiny_650e_coco.pdparams # 动态图模型路径
Note: For detailed tutorials on model conversion, please refer to
https://github.com/PaddlePaddle/PaddleSeg/blob/release/2.6/docs/model_export_cn.md
https://github.com/PaddlePaddle/PaddleClas/blob/release/2.4/README_ch.md
After the conversion is completed, we can get two sets of model.pdmodel and model.pdiparams files and corresponding yaml configuration files respectively.
Developed based on FastDeploy
OpenVINO reasoning module
This project uses the Intel NUC mini computer kit, and FastDeploy's OpenVINO backend is selected as the inference deployment solution. In the past, using OpenVINO required downloading packages, installation and configuration, and the process was relatively cumbersome. Therefore, I adopted the FastDeploy deployment solution and called its built-in OpenVINO inference backend for rapid development and deployment. For FastDeploy's precompiled library installation and usage tutorial, please refer to the official Github documentation.
- Link
https://github.com/PaddlePaddle/FastDeploy
Considering the applicability of the program and compatibility with various hardware environments, I first wrote the Option configuration to select different inference backends according to different hardware. In the CPU environment, OpenVINO is used as the inference backend by default.
import cv2import numpy as npimport fastdeploy as fdfrom PIL import Imagefrom collections import Counterdef FastdeployOption(device=0): option = fd.RuntimeOption() if device == 0: option.use_gpu() else: # 使用OpenVino推理 option.use_openvino_backend() option.use_cpu() return option
Then, the reasoning code of the semantic segmentation model is encapsulated into a class, which is convenient for the front-end to call quickly. In the init method, the SegModel() function is directly called to initialize the model (hot loading), and the reasoning of the results is completed through SegModel.predict(). After the reasoning results are obtained, postprocess() is executed to analyze the results and extract the pixels of buildings and green spaces. Quantity, count the proportion of images, and get the proportion result of environmental elements. Finally, call FastDeploy's built-in vis_segmentation() visualization function to visualize the inference results.
class SegModel(object): def __init__(self, device=0) -> None: self.segModel = fd.vision.segmentation.ppseg.PaddleSegModel( model_file = 'inference/ppliteseg/model.pdmodel', params_file = 'inference/ppliteseg/model.pdiparams', config_file = 'inference/ppliteseg/deploy.yaml', runtime_option=FastdeployOption(device) ) def predict(self, img): segResult = self.segModel.predict(img) result = self.postprocess(segResult) visImg = fd.vision.vis_segmentation(img, segResult) return result, visImg def postprocess(self, result): resultShape = result.shape labelmap = result.label_map labelmapCount = dict(Counter(labelmap)) pixelTotal = int(resultShape[0] * resultShape[1]) # 统计建筑率和绿地率 buildingRate, greenRate = 0, 0 if 8 in labelmapCount: buildingRate = round(labelmapCount[8] / pixelTotal* 100, 3) if 9 in labelmapCount: greenRate = round(labelmapCount[9] / pixelTotal * 100 , 3) return {"building": buildingRate, "green": greenRate}
In the same way, directly call the PPYOLO() method of FastDeploy to complete the reasoning of the model, and call the corresponding visualization function vis_detection() to render after post-processing and formatting the data.
class DetModel(object): def __init__(self, device=0) -> None: self.detModel = fd.vision.detection.PPYOLO( model_file = 'inference/ppyolo/model.pdmodel', params_file = 'inference/ppyolo/model.pdiparams', config_file = 'inference/ppyolo/infer_cfg.yml', runtime_option=FastdeployOption(device) ) # 阈值设置 self.threshold = 0.3 def predict(self, img): detResult = self.detModel.predict(img.copy()) result = self.postprocess(detResult) visImg = fd.vision.vis_detection(img, detResult, self.threshold, 2) return result, visImg def postprocess(self, result): # 得到结果 detIds = result.label_ids detScores = result.scores # 统计数量 humanNum, CarNum = 0, 0 for i in range(len(detIds)): if detIds[i] == 0 and detScores[i] >= self.threshold: humanNum += 1 if detIds[i] == 2 and detScores[i] >= self.threshold: CarNum += 1 return {"human": humanNum, "car": CarNum}
After encapsulating the PP-LiteSeg semantic segmentation model and PP-YOLO Tiny target detection model into a class, save it as inferEngine.py file for subsequent front-end code calls.
Combined with PySide6
Develop a visual GUI interface
The front-end development uses PySide6, and the source code of the interface can be obtained from the Github project link at the end of the article. On the whole, the development is not difficult. The main difficulty lies in the program freezing or delay caused by the simultaneous update of the three video playback components.
The solution applied in this project is to use multi-threading to separate the back-end of the three video playback components into three independent threads. One thread (displayThread) pushes the real-time video stream original painting to the front-end for update, and the other two threads (segThread and detThread) synchronously complete semantic segmentation and target detection inference real-time video frame images and update the inference result images after post-processing to the front-end components. The specific code is as follows (main.py).
Scroll down to see all content
Finally, execute python main.py to run the program and check the reasoning effect.
**
summary **
FastDeploy is an inference deployment toolbox that helps developers quickly deploy deep learning models. It has built-in backends including OpenVINO, TensorRT, ONNX Runtime, and Paddle Inference, and is integrating inference backends such as Paddle Lite and RKNN. Targeted optimization of each backend is well compatible with the hardware of the Intel NUC mini-host, no need to install the OpenVINO kit and configuration environment, and it also eliminates the trouble of tuning and speeding up, reducing the learning curve for developers cost and deployment cost, improving the efficiency of deployment and development. At present, it has supported 60+ popular ecological models including flying paddles. For more reasoning deployment of AI models, you can go to FastDeploy's GitHub.
- Link
https://github.com/PaddlePaddle/FastDeploy
Wonderful past
Follow 【Flying PaddlePaddlePaddle】public account
Get more technical content~