Machine Vision (4)-Machine Vision Application
table of Contents
One, face recognition
Face recognition needs to determine the position, size and posture of the face (if it exists) in the input image, and is often used in applications such as biometric recognition, video monitoring, and human-computer interaction. In 2001, Viola and Jones proposed a cascaded classifier object detection algorithm based on Haar features, which was improved by Lienhart and Maydt in 2002, providing an effective method for fast and reliable face detection applications. OpenCV has integrated the open source implementation of the algorithm, using the Haar features of a large number of samples for classifier training, and then calling the trained waterfall cascade classifier cascade for pattern matching.
The face recognition algorithm in OpenCV first converts the acquired image to grayscale, and performs edge processing and noise filtering; then the image is reduced, the histogram is equalized, and the matching classifier is enlarged by the same multiple until it matches the classifier. If the size is larger than the detected image, the matching result will be returned. In the matching process, the matching can be performed according to different types in the cascade classifier, such as front face and side face.
OpenCV has integrated the face recognition algorithm, only need to call the corresponding interface of OpenCV to realize the function of face recognition.
Source code realization
1. Initialization part The
initialization part mainly completes the setting of ROS nodes, images, and recognition parameters.
def __init__(self):
rospy.on_shutdown(self.cleanup);
# 创建cv_bridge
self.bridge = CvBridge()
self.image_pub = rospy.Publisher("cv_bridge_image", Image, queue_size=1)
# 获取haar特征的级联表的XML文件，文件路径在launch文件中传入
cascade_1 = rospy.get_param("~cascade_1", "")
cascade_2 = rospy.get_param("~cascade_2", "")
# 使用级联表初始化haar特征检测器
self.cascade_1 = cv2.CascadeClassifier(cascade_1)
self.cascade_2 = cv2.CascadeClassifier(cascade_2)
# 设置级联表的参数，优化人脸识别，可以在launch文件中重新配置
self.haar_scaleFactor = rospy.get_param("~haar_scaleFactor", 1.2)
self.haar_minNeighbors = rospy.get_param("~haar_minNeighbors", 2)
self.haar_minSize = rospy.get_param("~haar_minSize", 40)
self.haar_maxSize = rospy.get_param("~haar_maxSize", 60)
self.color = (50, 255, 50)
# 初始化订阅rgb格式图像数据的订阅者，此处图像topic的话题名可以在launch文件中重映射
self.image_sub = rospy.Subscriber("input_rgb_image", Image, self.image_callback, queue_size=1)
2. ROS image callback function After
receiving the RGB image data released by the camera, the routine node enters the callback function, converts the image into OpenCV data format, and then starts to call the face recognition function after preprocessing, and finally publishes the recognition result.
def image_callback(self, data):
# 使用cv_bridge将ROS的图像数据转换成OpenCV的图像格式
try:
cv_image = self.bridge.imgmsg_to_cv2(data, "bgr8")
frame = np.array(cv_image, dtype=np.uint8)
except CvBridgeError, e:
print e
# 创建灰度图像
grey_image = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)
# 创建平衡直方图，减少光线影响
grey_image = cv2.equalizeHist(grey_image)
# 尝试检测人脸
faces_result = self.detect_face(grey_image)
# 在opencv的窗口中框出所有人脸区域
if len(faces_result)>0:
for face in faces_result:
x, y, w, h = face
cv2.rectangle(cv_image, (x, y), (x+w, y+h), self.color, 2)
# 将识别后的图像转换成ROS消息并发布
self.image_pub.publish(self.bridge.cv2_to_imgmsg(cv_image, "bgr8"))
3. Face
recognition Face recognition directly calls the face recognition interface provided by OpenCV to match the facial features in the database.
def detect_face(self, input_image):
# 首先匹配正面人脸的模型
if self.cascade_1:
faces = self.cascade_1.detectMultiScale(input_image,
self.haar_scaleFactor,
self.haar_minNeighbors,
cv2.CASCADE_SCALE_IMAGE,
(self.haar_minSize, self.haar_maxSize))
# 如果正面人脸匹配失败，那么就尝试匹配侧面人脸的模型
if len(faces) == 0 and self.cascade_2:
faces = self.cascade_2.detectMultiScale(input_image,
self.haar_scaleFactor,
self.haar_minNeighbors,
cv2.CASCADE_SCALE_IMAGE,
(self.haar_minSize, self.haar_maxSize))
return faces
There are some parameters and topic names in the code that need to be set in the launch file, so you also need to write a launch file to run the routine.
<launch>
<node pkg="test2" name="face_detector" type="face_detector.py" output="screen">
<remap from="input_rgb_image" to="/usb_cam/image_raw" />
<rosparam>
haar_scaleFactor: 1.2
haar_minNeighbors: 2
haar_minSize: 40
haar_maxSize: 60
</rosparam>
<param name="cascade_1" value="$(find robot_vision)/data/haar_detectors/haarcascade_frontalface_alt.xml" />
<param name="cascade_2" value="$(find robot_vision)/data/haar_detectors/haarcascade_profileface.xml" />
</node>
</launch>
2. Object tracking
Object tracking and object recognition have similarities, and the same method of feature point detection is used, but the focus is different. The object for object recognition can be static or dynamic, and the model established based on the feature points of the object is used as the data basis for recognition; object tracking emphasizes the accurate positioning of the object position, and the input image generally needs to have dynamic characteristics.
The object tracking function first samples the feature points of the object in the current frame of the image according to the input image stream and the selected object to be tracked; then compares the gray value of the current frame and the next frame to estimate the characteristics of the tracked object in the current frame The position of the point in the next frame of image; then filter the feature points with the same position, the remaining points are the feature points of the tracking object in the second frame of image, and the feature point cluster is the location of the tracking object. This function is still based on the image processing algorithm provided by OpenCV.
Try to choose a test object with a solid color background and a large color difference. Move the recognized object in the screen, you can see that the rectangular frame identifies the real-time position of the moving object, and the recognition area, threshold and other parameters can be adjusted for the experimental environment.
Source code realization
1. Initialization part The
initialization part mainly completes the setting of ROS nodes, images, and recognition parameters.
def __init__(self):
rospy.on_shutdown(self.cleanup);
# 创建cv_bridge
self.bridge = CvBridge()
self.image_pub = rospy.Publisher("cv_bridge_image", Image, queue_size=1)
# 设置参数：最小区域、阈值
self.minArea = rospy.get_param("~minArea", 500)
self.threshold = rospy.get_param("~threshold", 25)
self.firstFrame = None
self.text = "Unoccupied"
# 初始化订阅rgb格式图像数据的订阅者，此处图像topic的话题名可以在launch文件中重映射
self.image_sub = rospy.Subscriber("input_rgb_image", Image, self.image_callback, queue_size=1)
2. After the image processing
routine node receives the RGB image data released by the camera, it enters the callback function to convert the image into OpenCV format; after the image preprocessing is completed, the two frames of images are compared, and the moving object is identified based on the image difference , To identify the recognition results and publish them.
def image_callback(self, data):
# 使用cv_bridge将ROS的图像数据转换成OpenCV的图像格式
try:
cv_image = self.bridge.imgmsg_to_cv2(data, "bgr8")
frame = np.array(cv_image, dtype=np.uint8)
except CvBridgeError, e:
print e
# 创建灰度图像
gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)
gray = cv2.GaussianBlur(gray, (21, 21), 0)
# 使用两帧图像做比较，检测移动物体的区域
if self.firstFrame is None:
self.firstFrame = gray
return
frameDelta = cv2.absdiff(self.firstFrame, gray)
thresh = cv2.threshold(frameDelta, self.threshold, 255, cv2.THRESH_BINARY)[1]
thresh = cv2.dilate(thresh, None, iterations=2)
binary, cnts, hierarchy= cv2.findContours(thresh.copy(), cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE)
for c in cnts:
# 如果检测到的区域小于设置值，则忽略
if cv2.contourArea(c) < self.minArea:
continue
# 在输出画面上框出识别到的物体
(x, y, w, h) = cv2.boundingRect(c)
cv2.rectangle(frame, (x, y), (x + w, y + h), (50, 255, 50), 2)
self.text = "Occupied"
# 在输出画面上打当前状态和时间戳信息
cv2.putText(frame, "Status: {}".format(self.text), (10, 20),
cv2.FONT_HERSHEY_SIMPLEX, 0.5, (0, 0, 255), 2)
# 将识别后的图像转换成ROS消息并发布
self.image_pub.publish(self.bridge.cv2_to_imgmsg(frame, "bgr8"))
There are some parameters and topic names in the code that need to be set in the launch file, so you also need to write a launch file to run the routine.
<launch>
<node pkg="test2" name="motion_detector" type="motion_detector.py" output="screen">
<remap from="input_rgb_image" to="/usb_cam/image_raw" />
<rosparam>
minArea: 500
threshold: 25
</rosparam>
</node>
</launch>
Three, QR code recognition
Two-dimensional codes are used in more and more scenes in life. Whether it is shopping in malls or sharing bicycles, two-dimensional codes have been widely used as an entrance sign. ROS provides a variety of QR code recognition function packages-(ar_track_alvar)
1. ar_track_alvar function package
The installation of the function package ar_track_alvar is very simple, just use the following command:
sudo apt-get install ros-kinetic-ar-track-alvar
After the installation is complete, find the ar_track_alvar function package in the default installation directory of ROS. Open the launch folder under the function package, you can see multiple launch files. These are the QR code recognition routines used by the PR2 robot, which can be modified on the basis of these files, so that your machine vision has the function of QR code recognition.
2. Create QR code The
ar_track_alvar function package provides the function of generating QR code labels. You can use the following commands to create QR code labels with corresponding labels:
rosrun ar_track_alvar createMarker AR_ID
The AR_ID can be any number from 0 to 65535, for example:
rosrun ar_track_alvar createMarker 0
You can create a QR code label picture with a label of 0, name it MarkerData_0.png, and place it under the current path of the terminal. You can also directly use the system picture viewer to open the QR code file.
The createMarker tool also has many parameters that can be configured. Use the following command to see the help.
rosrun ar_track_alvar createMarker
createMarker can not only use digital labels to generate QR code labels, but also use strings, file names, URLs, etc., and can also use the -s parameter to set the size of the generated QR code.
You can use the following commands to create a series of QR code labels:
roscd robot_vision/config
rosrun ar_track_alvar createMarker -s 5 0
rosrun ar_track_alvar createMarker -s 5 1
rosrun ar_track_alvar createMarker -s 5 2
3. The camera recognition QR code
ar_track_alvar function package supports USB camera or RGB-D camera as a visual sensor to recognize the QR code, corresponding to the two different recognition nodes of individualNoKinect and individualMarkers respectively.
4. Kinect recognizes two
-dimensional codes. RGB-D cameras such as Kinect can also be used to recognize two-dimensional codes.
Fourth, object recognition
ROS integrates a powerful object recognition framework-Object Recognition Kitchen (ORK), which contains a variety of three-dimensional object recognition methods.
1. ORK function package
The operation is performed under Ubuntu 16.04. Since this version of kinetic does not integrate all ORK function packages, the following steps are required to install the source code.
First install the dependent libraries using the following command:
sudo apt-get install meshlab
sudo apt-get install libosmesa6-dev
sudo apt-get install python-pyside.qtcore
sudo apt-get install python-pyside.qtgui
Download the rosinstall file and compile the
file link: https://github.com/wg-perception/object_recognition_core
Create a new ork_ws workspace, download the ork.rosinstall.kinetic.plus file directly to ork_ws, and then proceed as follows:
cd ork_ws
wstool init src [文件路径]/ork.rosinstall.kinetic.plus
Download function source code:
cd src
wstool update -j8
git clone https://github.com/jbohren/xdot.git
cd ..
rosdep install --from-paths src -i -y
Compile:
cd ork_ws
catkin_make
Object recognition process:
(1) Create an object model that needs to be recognized
(2) Train the model to generate a recognition model
(3) Use the trained recognition model for object recognition
2. Build an object model library
Create a database
Install CouchDB tools:
sudo apt-get install couchdb
Test whether the installation is successful:
curl -X GET http://localhost:5984
Create a piece of Coke model data:
rosrun object_recognition_core object_add.py -n "coke " -d "A universal can of coke " --commit
Load the 3D model
Download the existing coke.stl model and download it via github:
git clone https://github.com/wg-perception/ork_tutorials
Download to the src file;
then load the Coke model into the database:
rosrun object_recognition_core mesh_add.py 49cce25ad1745bc78d8c16a90200008e [path]/ork_tutorials/data/coke.stl --commit
View the model:
sudo pip install git+https://github.com/couchapp/couchapp.git
rosrun object_recognition_core push.sh
Click object_listing:
click meshes:
3. Model training
There are many models loaded in the database, and we need to train to generate matching templates.
The command is as follows:
sudo object_recognition_core training -c `rospack find object_recognition_linemod`/conf/training.ork
4. Three-dimensional object recognition