Do together ROS-DEMO series (2): matching target recognition based find_object_2d

Copyright: Attribution, allow others to create paper-based, and must distribute paper (based on the original license agreement with the same license Creative Commons )

• Connect article, we hope to be more intelligent robots that get hold of anything we want, rather than through labeling (ar_makrer) or simple color filter division (for example, to identify a solid fixed object) to identify the target object. So we intend to use other methods to identify the target identification.
• Currently I know of a better way to get started, there are two, one is the use of template matching (including color images or point cloud matching), the second is to use the nearest hot machine learning to do (such as yolo, etc., would later introduce specific). So this time I'll talk about the rapid deployment based on template matching color image, because the current ROS existing find_object_2d feature pack allows us to achieve without the need to understand the specific principles, quick start this feature, more suitable for the task .
• Reference:
  http://wiki.ros.org/find_object_2d
  https://github.com/introlab/find-object/tree/kinetic-devel
  http://introlab.github.io/find-object/
  

---

---

1, find_object_2d function package installed

Ros hanging directly corresponds to the version can be installed, install other versions see the official description: https://github.com/introlab/find-object/tree/kinetic-devel

# ROS Kinetic:
 $ sudo apt-get install ros-kinetic-find-object-2d
# ROS Jade:
 $ sudo apt-get install ros-jade-find-object-2d
# ROS Indigo:
 $ sudo apt-get install ros-indigo-find-object-2d
# ROS Hydro:
 $ sudo apt-get install ros-hydro-find-object-2d

---

2, the use of visual function packages

The package has two function nodes find_object_2dand find_object_3dwherein find_object_3dis prepared for kinect depth camera or the like, by three-dimensional coordinates of the center of the recognition target matching the target depth information output destination.

2.1 、find_object_2d

Boot process:

 $ roscore 
 $ 启动你的相机节点，输出图像话题
 $ rosrun find_object_2d find_object_2d image:=image_raw

Note image: = image_raw, this is a topic remapping, image_raw here to modify your camera to the subject image output.

• launch files written:

<launch>
	<!-- Nodes -->
	<node name="find_object_2d" pkg="find_object_2d" type="find_object_2d" output="screen">
		<remap from="image" to="image"/>
		<param name="gui" value="false" type="bool"/>
		<param name="objects_path" value="~/objects" type="str"/>
		<param name="settings_path" value="~/.ros/find_object_2d.ini" type="str"/>
	</node>
</launch

2.2 、find_object_3d

• :( boot process due to the need to call multiple parameters, launch file configuration parameters used here, start the node)

• 1, start the depth camera, such as:

# kinect2
roslaunch kinect2_bridge kinect2_bridge,launch

# realsesnse D414/435
roslaunch realsense2_camera rs_camera.launch

• 2, start find_object_3d identification:

<launch>
	<!-- Example finding 3D poses of the objects detected -->
	<!-- $ roslaunch kinect2_bridge kinect2_bridge.launch publish_tf:=true -->

	<!-- Which image resolution: sd, qhd, hd -->
	<arg name="resolution" default="qhd" />
	
	<node name="find_object_3d" pkg="find_object_2d" type="find_object_2d" output="screen">
		<param name="gui" value="true" type="bool"/>
		<param name="settings_path" value="~/.ros/find_object_2d.ini" type="str"/>
		<param name="subscribe_depth" value="true" type="bool"/>
		<param name="objects_path" value="" type="str"/>
		<param name="object_prefix" value="object" type="str"/>
		
		<remap from="rgb/image_rect_color" to="/kinect2/$(arg resolution)/image_color_rect"/>
		<remap from="depth_registered/image_raw" to="/kinect2/$(arg resolution)/image_depth_rect"/>
		<remap from="depth_registered/camera_info" to="/kinect2/$(arg resolution)/camera_info"/>
	</node>
</launch>

---

3, the specific use

Whether starting 2d, 3d which node, there will be a visual interface of a QT.

3.1 adjustment parameter, and the algorithm selection features operator

You may be selected SURF, SIFT, ORB, FAST series of algorithms, and to observe the effect on the number of feature points of the target object, and fluency processing measure to select the appropriate algorithm. Through the menu View - Parameters tab bar to bring up the parameters to be adjusted.
Figure:

3.2 Select the object matching, matching

In the left column, right option to add a new target, select the target by marquee way.

Then you can see the target object is a marquee success.

3.3 access to relevant topics

3.3.1 node receives the topic

//find_object_2d节点使用
topic：image (message_type: sensor_msgs/Image) 

//find_object_3d节点使用
topic：rgb/image_rect_color (message_type: sensor_msgs/Image) //对齐后的彩色图像

topic：rgb/camera_info (message_type: sensor_msgs/CameraInfo) //相机标定信息

topic：registered_depth/image_raw (message_type: sensor_msgs/Image)//深度图像

Node 3.3.2 release topic

topic：objects (message_type: std_msgs/Float32MultiArray)
//话题内容按照以下方式排列 [objectId1, objectWidth, objectHeight, h11, h12, h13, h21, h22, h23, h31, h32, h33, objectId2...] 
 //hxx 是 3x3 矩阵 (h31 = dx and h32 = dy, see QTransform)

topic：objectsStamped (message_type: find_object_2d/ObjectsStamped)
//带标签的对象物体

• Note: Also if we use find_object_3d, you can open rviz visualization tool, see the target's coordinates TF has been issued to the (here needs the correct configuration of TF camera coordinate system).
• The topic /objectsof the matrix h is not conducive to understanding, here the official also provides another node for us to learn:
  rosrun find_object_2d print_objects_detected image:=/camera/color/image_raw
• Observing the code can see the node processing node matrix h coordinate, id and the target output frame. Source see here

此节点的输出：
Object 3 detected, Qt corners at (786.949707,-11.180605) (1497.147252,-69.618191) (813.221906,428.768064) (1425.264906,463.355441)

printf("Object %d detected, Qt corners at (%f,%f) (%f,%f) (%f,%f) (%f,%f)\n",
					id,
					qtTopLeft.x(), qtTopLeft.y(),
					qtTopRight.x(), qtTopRight.y(),
					qtBottomLeft.x(), qtBottomLeft.y(),
					qtBottomRight.x(), qtBottomRight.y());

In addition to this node also released images of the target coordinates of the center, you can rqt_image_viewsee /image_with_objects.

3.3 Save configuration file and template matching

Click on File in the upper left corner, holds the matching template and configuration, and is placed under their own corresponding directory, as I have kept to the config folder, use the right to read their launch files.

<launch>
	<!-- My finding 3D poses of the objects detected -->
	<node name="find_object_3d" pkg="find_object_2d" type="find_object_2d" output="screen">
		<param name="gui" value="true" type="bool"/>

		<param name="settings_path" value="~/dhrobot_ws/src/dhrobot_demo/config/find_object_2d.ini" type="str"/>
		<param name="subscribe_depth" value="true" type="bool"/>
		<param name="objects_path" value="~/dhrobot_ws/src/dhrobot_demo/config" type="str"/>
		<param name="object_prefix" value="object" type="str"/>

		<remap from="rgb/image_rect_color" to="/camera/color/image_raw"/>
		<remap from="depth_registered/image_raw" to="/camera/depth/image_rect_raw"/>
		<remap from="depth_registered/camera_info" to="/camera/depth/camera_info"/>
	</node>
</launch>

4, the coordinates of the robot arm to a point identified

When using find_object_3d, we can directly obtain tf coordinates of the target object, so you can use tf ros comes directly converted to standard query manipulator base tf relationship of the object, and send the robot arm:
Reference here: HTTP: // docs.ros.org/lunar/api/tf/html/c++/classtf_1_1TransformListener.html
http://wiki.ros.org/tf/Tutorials/Writing%20a%20tf%20listener%20%28C%2B%2B%29

tf::StampedTransform transform;
  25     try{
  26       listener.lookupTransform("/arm_base_link", "/object_1",  
  27                                ros::Time(0), transform);
  28     }
  29     catch (tf::TransformException ex){
  30       ROS_ERROR("%s",ex.what());
  31       ros::Duration(1.0).sleep();
  32     }
  33 

5, the function package Evaluation

• Overall, this feature pack installation is fairly easy, but because it is the template matching, so the limitations of relatively large, not being the case of camera recognition success rate is not high on the target object, and want to get good results need to be carefully matched tuning parameters, a suitable algorithm rotation, center point coordinates of the output of the additional tf substantially accurate, but the attitude is not very reliable.