Automatic identification and localization of tomatoes based on Kinect vision system

Dissertation key notes

• 1. Tomato recognition based on image segmentation
  • Choice of color space
    • Disadvantages of segmentation using RGB color space
      • The background noise cannot be completely removed, the three components are highly correlated, and the segmentation result is greatly affected by the illumination.
    • RGB linear transformation
      • YIQ 、 YUV 、 I1I2I3
      • Advantages: small amount of calculation, and partially eliminate the correlation of RGB
    • RGB nonlinear transformation
      • HSI、 Lab、 Luv、 NRGB
    • Selection: HSI and Lab color spaces are divided separately, then the two segmentation results are fused, and the background noise is removed from the segmentation results.
  • HSI color feature analysis and extraction
    • HSI color feature analysis
      • Color space conversion formula

        

      • H stands for hue; S for saturation; and I for lightness or transparency. In the HSI color space, the three components of H, S, and I are independent of each other, and these three quantities can be controlled separately. The surface of tomatoes is smooth, which affects the extraction of images in the case of illumination changes, so only the H component is considered during processing, which avoids the influence of illumination. The specific method is to use the threshold-based improved Otsu method (threshold segmentation) to complete the H component. Image segmentation.
    • Improvement of Otsu segmentation method
      • Combine the iterative method and the Otsu method, first determine an initial threshold T1 according to the image, and then use the Otsu method to automatically obtain the optimal threshold T, and compare the two obtained thresholds. If T1≥T, then the optimal threshold is T, if T1 ≤T, the optimal threshold is T1.
    • segmentation result

      • 

  • Lab color space image segmentation
    • Color space conversion formula

      

    • The Lab color space is a 3-dimensional space map composed of chromaticity and luminance, where L represents the psychological lightness, and a and b are the psychological chromaticity. In the Lab color space coordinates, +a means red, -a means green, +b means yellow, -b means blue, and the lightness of the color is represented by the percentage of L. From the color characteristics of tomatoes, it is obvious that ripe tomatoes are +a (red), and immature tomatoes with stems and leaves are -a (green). Therefore, it is very suitable to choose the Lab color space to segment ripe tomatoes.
    • Separate the images of the L channel, a channel, and b channel of the image to obtain 3 independent grayscale images. Since the colors represented by the two ends of the a channel are the red of the ripe tomatoes and the background color green, only the image of the a channel needs to be processed to complete the segmentation of the ripe tomatoes. There are many segmentation methods based on grayscale images, and K-means clustering algorithm is used.
    • segmentation result

      • 

  • Fusion of HSI and Lab color space segmentation results
    • Image fusion method
      • Methods: color transition method, wavelet fusion method and multi-resolution spline method, etc.
      • Pros and cons: These algorithms perform better, but require more computation
    • Weighted Average Algorithm (this article)
      • Algorithm idea: The gray value P of the pixel point in the overlapping area of ​​the image is obtained by the weighted average of the gray value PH and Pa of the corresponding point in the two images.
    • denoising
      • Morphological operation is adopted to remove the background noise in the small connected area;
      • For the background noise with a large connected area, a threshold is used to determine the area.
    • result

      • 

  • Complicated Situation Research Discussion
    • Occluded by branches and leaves, only a part of the image of tomatoes can be collected——Method of generalized Hough transform
    • Two tomatoes are connected together——Segmentation algorithm based on boundary tracking
• 2. Kinect-based visual positioning
  • Introduction to Kinect
    • To realize the automatic picking of tomatoes by the robot, the key is to obtain the accurate spatial position of the tomatoes in order to provide parameters for controlling the action of the manipulator.
    • Contains RGB color camera, infrared transmitter, infrared sensor. The RGB color camera is used to capture color images. The depth image information is acquired actively. The infrared transmitter emits infrared rays, and the infrared sensor receives the reflected infrared rays. By processing the reflected information, the information of each pixel can be obtained. depth value.
  • 3D reconstruction and object localization
    • Use Kinect to obtain RGB color information and depth information;
    • Segment the color image, analyze the connected area in the segmentation result, and determine the number of tomatoes and the two-dimensional coordinates of the pixels corresponding to each tomato;
    • According to the formula, the two-dimensional coordinates of the centroid of each tomato are obtained;

      

    • According to the depth value of the corresponding pixel in the depth image obtained by Kinect, the three-dimensional coordinates of each tomato centroid are calculated by using the reprojectimageTo3D function of OpenCV.
• 3. Conclusion
  • The problem of automatic identification of tomatoes is analyzed, and it is proposed to divide the tomato according to the color channel, and integrate the segmentation results of HSI and Lab to complete the identification of tomatoes. Segmented out, the outline is clear and the image is full.
  • Using Kinect instead of binocular camera, 3D reconstruction of the depth image was carried out, and the spatial coordinate value of the target tomato centroid was obtained to complete the positioning, and an experimental platform was built to conduct the positioning test of the tomato target in the model.
  • The identification and localization method proposed in this paper is not only suitable for tomatoes, but can be used for automatic identification of red vegetables or fruits such as strawberries and cherries with a slight modification.