Halcon image stitching

1. Why splicing
   If your subject is small enough;
   if your lens field of view is large enough;
   if you have enough money, you can buy a better camera and a better lens. . .

If you don't have so many ifs, there are too many projects, and image stitching is inevitable.

2. What is the effect?
   With the help of Halcon's own example, the following two images are stitched into a wider image.

Image 1:
Write the picture description here

Image 2:
Write the picture description here

Image after stitching:
write the picture description here

Has it become wider?

3. Mosaic steps
   Read image
   Extract feature points
   Calculate transformation matrix
   Mosaic
   Refer to the Halcon routine proj_match_points_distortion_ransac.hdev, and analyze step by step. This routine stitches images based on feature points.
   Write picture description here

3.1. Read the image and display the image
code:

read_image (Image1, ‘building_01’)
read_image (Image2, ‘building_02’)
get_image_size (Image1, Width, Height)
dev_close_window ()
dev_open_window (0, 0, Width, Height, ‘white’, WindowHandle)
set_display_font (WindowHandle, 16, ‘mono’, ‘true’, ‘false’)
dev_display (Image1)
disp_message (WindowHandle, ‘Image 1 to be matched’, ‘image’, -1, -1, ‘black’, ‘true’)
disp_continue_message (WindowHandle, ‘black’, ‘true’)
stop ()
dev_display (Image2)
disp_message (WindowHandle, ‘Image 2 to be matched’, ‘image’, -1, -1, ‘black’, ‘true’)
disp_continue_message (WindowHandle, ‘black’, ‘true’)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
Effect:
write a picture description here

Write picture description here

3.2. Obtaining feature points
This routine stitches images based on the feature points of the image, and it needs to obtain the feature points of two images.
What are feature points? According to the explanation of the operator points_foerstner, there are two types of feature points, one is called intersection feature points, which refer to the points on the edge of the image, and the other is called regional feature points, for example, the color and brightness of the image are different from the surrounding ones. point.

Code:

points_foerstner (Image1, 1, 2, 3, 50, 0.1,'gauss','true', Rows1, Columns1, CoRRJunctions, CoRCJunctions, CoCCJunctions, RowArea, ColumnArea, CoRRArea, CoRCArea, CoCCArea)
points_foerstner (Image2, 1, 2, 3, 50, 0.1,'gauss','true', Rows2, Columns2, CoRRJunctions, CoRCJunctions, CoCCJunctions, RowArea, ColumnArea, CoRRArea, CoRCArea, CoCCArea)
1
2
3
3.3. Calculate the affine transformation matrix
based on the characteristics of the two images Point, calculate the affine transformation matrix.

Code:

proj_match_points_ransac (Image1, Image2, Rows1, Columns1, Rows2, Columns2,'ncc', 10, 0, 0, Height, Width, 0, 0.5,'gold_standard', 2, 42, HomMat2DUnrectified, Points1Unrectified, Points2Unrectified)
1
3.3.
According to the affine transformation matrix for splicing.

Code:

concat_obj (Image1, Image2, Images)
gen_projective_mosaic (Images, MosaicImageUnrectified, 1, 1, 2, HomMat2DUnrectified,'default','false', MosaicMatrices2DUnrectified)
1
2
3
Effect:
write the picture description here

Show seam code:

projective_trans_pixel (MosaicMatrices2DUnrectified[9:17], [0,493], [0,0], RowTrans, ColumnTrans)
gen_contour_polygon_xld (Contour, RowTrans, ColumnTrans)
1
2
3 Joint
effect:
write the picture description here

Carefully observe the seams where the images are spliced, and find that the splicing effect is not ideal and the seams are staggered. The reason is the radial distortion of the two images. What is radial distortion? This is inherent to the lens. When the focal length is very large or very small, the captured image is especially obvious. The edge of the image is concave forward or convex inward. Carefully observe the original two images, the edge is convex inward Go in.

Extracurricular knowledge In-
camera parameters:

f: the principal moment of the camera, that is, the focal length
k: the size of radial distortion, that is, radial distortion, generally does not consider tangential distortion
sx, sy: the distance between adjacent pixels in the horizontal and vertical directions of the image sensor
cx, cy : The vertical projection of the projection center on the imaging plane
1
2
3
4
External camera parameters:

Translation vector X, Y, Z
rotation vector X, Y, Z
perspective correction
1
2
3
The internal and external parameters of the camera are the focus of camera calibration.

Therefore, the second half of the routine is to eliminate the influence of this radial distortion on splicing. There are corresponding operators in Halcon, which is very convenient to use.

4. Eliminate radial distortion
   4.1. Read out the image
   Same as above

4.2. Calculate the affine transformation matrix
Note that an operator that eliminates radial distortion is used. Code:

proj_match_points_distortion_ransac (Image1, Image2, Rows1, Columns1, Rows2, Columns2,'ncc', 10, 0, 0, Height, Width, 0, 0.5,'gold_standard', 1, 42, HomMat2D, Kappa, Error, Points1, Points2)
1
4.3. Preprocessing
Eliminate mirror distortion in the image.

Code:

CamParDist: = [0.0, Kappa, 1.0,1.0,0.5 * (Width - 1), 0.5 * (Height - 1), Width, Height]
change_radial_distortion_cam_par ('fixed', CamParDist, 0, CamPar)
change_radial_distortion_image (Image1, Image1, Image1Rect, CamParDist, CamPar)
change_radial_distortion_image (Image2, Image2, Image2Rect, CamParDist, CamPar)
1
2
3
4
5
效果 ：

Image 1 (original image)
Write the description here

Image 1 (after removing the radial distortion)
write a picture description here

Image 2 (original image)
Write the description here

Image 2 (After removing the radial distortion),
write the description of the picture here. The
effect of the first group of pictures is not very obvious. Carefully observe the second group of pictures . Are the two edges flattened?

4.4. Image stitching
Same as above.

The final effect:
write a picture description here

Carefully observe the joints, this time the images are stitched together well.

5. Code The
   complete code is as follows:

• This example shows how to use proj_match_points_distortion_ransac to

• match two images in a mosaicking application.

• This example shows how to use the proj_match_points_distortion_ransac operator in splicing applications

• Mosaic two pictures (spliced ​​images based on feature point matching)

• Initialization

• Initialize
  dev_update_off ()

• Read and display the images

• 读取并显示图像
  read_image (Image1, ‘building_01’)
  read_image (Image2, ‘building_02’)
  get_image_size (Image1, Width, Height)
  dev_close_window ()
  dev_open_window (0, 0, Width, Height, ‘white’, WindowHandle)
  set_display_font (WindowHandle, 16, ‘mono’, ‘true’, ‘false’)
  dev_display (Image1)
  disp_message (WindowHandle, ‘Image 1 to be matched’, ‘image’, -1, -1, ‘black’, ‘true’)
  disp_continue_message (WindowHandle, ‘black’, ‘true’)
  stop ()
  dev_display (Image2)
  disp_message (WindowHandle, ‘Image 2 to be matched’, ‘image’, -1, -1, ‘black’, ‘true’)
  disp_continue_message (WindowHandle, ‘black’, ‘true’)
  stop ()

• Extract points to be matched from the images

• 获取特征点
  points_foerstner (Image1, 1, 2, 3, 50, 0.1, ‘gauss’, ‘true’, Rows1, Columns1, CoRRJunctions, CoRCJunctions, CoCCJunctions, RowArea, ColumnArea, CoRRArea, CoRCArea, CoCCArea)
  points_foerstner (Image2, 1, 2, 3, 50, 0.1, ‘gauss’, ‘true’, Rows2, Columns2, CoRRJunctions, CoRCJunctions, CoCCJunctions, RowArea, ColumnArea, CoRRArea, CoRCArea, CoCCArea)

• We will first perform a normal projective matching that does not take

• the radial distortions into account to show the errors that are caused

• by neglecting the radial distortions.

• First of all, without considering the radial distortion, perform image stitching, we will see the influence of the radial distortion

• The stitching effect at the seams is not ideal
  proj_match_points_ransac (Image1, Image2, Rows1, Columns1, Rows2, Columns2,'ncc', 10, 0, 0, Height, Width, 0, 0.5,'gold_standard', 2, 42, HomMat2DUnrectified , Points1Unrectified, Points2Unrectified)

• Construct a projective mosaic from the two unrectified images.

• Construct two unmodified (with radial distortion) images into a projective mosaic
  concat_obj (Image1, Image2, Images)
  gen_projective_mosaic (Images, MosaicImageUnrectified, 1, 1, 2, HomMat2DUnrectified,'default','false ', MosaicMatrices2DUnrectified)

• Display unrectified results

• 显示结果
  get_image_size (MosaicImageUnrectified, Width, Height)
  dev_set_window_extents (-1, -1, Width, Height)
  dev_clear_window ()
  dev_display (MosaicImageUnrectified)

• Display seam line

• 显示拼接缝隙
  projective_trans_pixel (MosaicMatrices2DUnrectified[9:17], [0,493], [0,0], RowTrans, ColumnTrans)
  gen_contour_polygon_xld (Contour, RowTrans, ColumnTrans)
  set_line_style (WindowHandle, [1,5])
  dev_set_line_width (1)
  dev_set_color (‘yellow’)
  dev_display (Contour)
  set_line_style (WindowHandle, [])
  dev_set_draw (‘margin’)
  dev_set_color (‘red’)
  dev_set_line_width (3)
  gen_circle (Circle, [82,402], [228,223], [15,15])
  dev_display (Circle)

• From the results, without considering the radial distortion, the splicing effect at the seams is not ideal, and the seams are staggered
  Message :='The mosaic image does not fit'
  Message[1] :='perfectly, if radial distortions'
  Message[2] :='are not taken into account.'
  disp_message (WindowHandle, Message,'image', 200, 300,'black','true')
  disp_continue_message (WindowHandle,'black','true' )
  stop ()

• Now, we will perform a projective matching that takes the radial

• distortions into account.

• This time, remove the influence of radial distortion, and perform stitching again
  get_image_size (Image1, Width, Height)
  proj_match_points_distortion_ransac (Image1, Image2, Rows1, Columns1, Rows2, Columns2,'ncc', 10, 0, 0, Height, Width, 0 , 0.5,'gold_standard', 1, 42, HomMat2D, Kappa, Error, Points1, Points2)

• Construct camera parameters for the purpose of rectifying the images,

• i.e., to remove the radial distortions.

• In order to modify the image, construct the camera parameters

• For example, to remove the radial distortion
  CamParDist := [0.0,Kappa,1.0,1.0,0.5 * (Width-1),0.5 * (Height-1),Width,Height]

• Remove the radial distortions from the images.

• 畸变 图像 d 的 径向 畸变
  change_radial_distortion_cam_par ('fixed', CamParDist, 0, CamPar)
  change_radial_distortion_image (Image1, Image1, Image1Rect, CamParDist, CamPar)
  change_radial_distortion_image (Image2, Image2, ImageP, ImagePDR),

• Construct a mosaic from the two rectified images. Note that the images

• fit together perfectly.

• Construct a stitched image using images with radial distortion removed

• You can see that the image stitching is very good
  concat_obj (Image1Rect, Image2Rect, ImagesRect)
  gen_projective_mosaic (ImagesRect, MosaicImage, 1, 1, 2, HomMat2D,'default','false', MosaicMatrices2D)

• Display rectified results

• Display the modified result
  get_image_size (MosaicImage, Width, Height)
  dev_set_window_extents (-1, -1, Width, Height)
  dev_clear_window ()
  dev_display (MosaicImage)

• Display seam line

• 显示接缝
  projective_trans_pixel (MosaicMatrices2D[9:17], [0,493], [0,0], RowTrans, ColumnTrans)
  gen_contour_polygon_xld (Contour2, RowTrans, ColumnTrans)
  set_line_style (WindowHandle, [1,5])
  dev_set_line_width (1)
  dev_set_color (‘yellow’)
  dev_display (Contour2)
  set_line_style (WindowHandle, [])
  dev_set_draw (‘margin’)
  dev_set_color (‘green’)
  dev_set_line_width (3)
  gen_circle (Circle, [124,496], [244,239], [15,15])
  dev_display (Circle)
  Message := ‘The mosaic image fits perfectly,’
  Message[1] := ‘if radial distortions are taken’
  Message[2] := ‘into account.’
  disp_message (WindowHandle, Message, ‘image’, 200, 300, ‘black’, ‘true’)

• Output the spliced ​​image
  write_image(MosaicImage,'bmp', 0,'result.bmp')