OpenCV digital image processing - detect geometric shapes in images and measure side lengths, diameters, and internal angles

1. Introduction

In traditional automated production dimensional measurement, a common method is to use calipers or micrometers to measure a certain parameter of the workpiece being measured multiple times and take the average of these measurement values. However, there are some problems with these traditional detection equipment or manual measurement methods: low measurement accuracy, slow measurement speed, and the inability to process measurement data in a timely manner. These limitations make it impossible to meet the needs of large-scale automated production.

In contrast, dimensional measurement methods based on machine vision technology have the following advantages:

1. Low cost: Machine vision equipment costs less than traditional equipment, and in some cases, ordinary cameras can be used.
2. High precision: Machine vision systems can provide high-precision measurement results, reaching micron-level accuracy.
3. Non-contact: No physical contact with the object being measured is required, thus avoiding damage or deformation of the object.
4. Real-time: Measurement results can be obtained in real time, improving production efficiency and finding problems in time.
5. Flexibility: The machine vision system can adapt to workpieces of different shapes and sizes, and has strong adaptability and flexibility.

In the automated manufacturing industry, machine vision technology is widely used in workpiece size measurement. Through the machine vision system, multiple dimensional parameters such as length, diameter, angle, and curvature of the workpiece can be measured, and even the basic geometric features of the relevant areas of the product can be detected. This technology can not only obtain the dimensional parameters of products in real time, but also perform online real-time judgment and sorting, which plays an important role in automated production.

Applications based on machine vision have covered the following areas:

1. Medical imaging

• Measure tumor or organ size: Use medical image processing technology to measure the size of tumors or organs in medical images.

2. Engineering and Manufacturing

• Detecting part dimensions: Detect the dimensions and defects of workpieces in manufacturing to ensure product quality and accuracy.
• Assembly and Positioning: Use image processing technology for assembly and positioning to ensure correct placement and alignment of parts.

3. Geographic Information Systems (GIS) and Remote Sensing

• Measuring surface features: In GIS, remote sensing images are used to measure the area and distribution of surface features.
• Topography: Measure terrain height and relief using digital elevation models (DEM) and remote sensing technology.

4. Agricultural and Environmental Sciences

• Vegetation Analysis: Analyze and evaluate crop growth and land use.
• Land use measurement: Use image processing techniques to measure farmland, forest cover, and land use types.

5. Machine Vision and Automation

• Product Inspection and Measurement: Use image processing technology to detect and measure product size and position in automated manufacturing.
• Navigation and positioning: Use images to perceive the environment and perform positioning in autonomous driving and robot navigation.

This article does not cover the selection of cameras and lenses, camera lens angle (horizontal viewing angle, vertical viewing angle, diagonal) error, camera installation height, detection field of view, measurement accuracy, light source and filter selection, etc. Equipment knowledge. If you want to discuss these hardware topics in depth, you can send a private message to the blogger.

2. Measurement process and principle

1. Project process

Let’s first look at the overall project implementation process. The most important part of the entire process should be target segmentation. If it is not segmented well, it will be meaningless to do any subsequent processing. The target can be segmented using traditional image processing, or Implemented through semantic segmentation based on deep learning.

The difference between deep learning and traditional image processing methods:

1. Feature extraction method:

• Traditional image processing: Traditional methods mainly rely on manually designed feature extractors, such as edge detectors, filters, etc. These methods often require expertise in number processing to select and design appropriate feature extraction methods.

• Deep learning: Deep learning models can automatically learn optimal feature representations from data. Deep learning architectures such as convolutional neural networks (CNN) can learn abstract features in images without manually designing feature extractors.

2. Data requirements:

• Traditional image processing: Traditional methods for feature extraction and processing usually require predefined knowledge about specialized fields and sometimes require large amounts of manually annotated data.

• Deep Learning: Deep learning methods have a more significant need for large-scale data sets, especially a large amount of labeled data to train complex deep neural networks.

3. Versatility and flexibility:

• Traditional image processing: Traditional methods are usually designed for specific problems, so they may not be versatile on other problems and have poor scene generalization capabilities.

• Deep Learning: Deep learning models can better generalize to unseen data and demonstrate greater versatility and flexibility in different fields.

4. Tuning and complexity:

• Traditional image processing: Traditional methods usually require manual adjustment of parameters and the design of feature extractors, which requires coders to have professional image processing knowledge and experience.

• Deep Learning: Deep learning models usually have more hyperparameters to tune and are relatively more complex. Therefore, training deep learning models may require more computing resources and time.

5. Scope of application:

• Traditional image processing: Traditional methods still have advantages in certain specific tasks, such as simple image filtering, edge detection and other fields.

• Deep learning: Deep learning has made major breakthroughs in many fields, such as image classification, object detection, segmentation, generative adversarial networks, etc., making it possible to achieve better results on complex problems. Performance.

2. Length measurement

To measure line segments in an image, you must first understand "pixels per metric ratio", which is similar to a scale. By knowing the size of an object on the image and the number of pixels the object occupies in the image, you can Get a proportional relationship so that the pixels of other objects can be converted into actual measurement units (such as centimeters, millimeters, etc.).

Key attributes include:

1. Known length: You need to know the actual length of an object in the image, usually expressed in some measurable unit (such as millimeters, inches, etc.).
2. Number of pixels: The number of pixels occupied by the object of known length in the image. This can be obtained by measuring the pixel width or height of that object in the image.

With these two attributes, the number of pixels corresponding to each unit of measurement can be calculated. This ratio converts the pixels in the image into actual units of measurement, allowing the size or length of other objects to be measured.

"Known length in pixels" is the number of pixels occupied by the known length reference object in the image, and "Known length in metrics" is the actual size of the known length reference object.

3. Angle measurement

In angle measurement, we must first clarify a theorem. The geometric shape and the angle therein are independent of the proportion of the image. In geometry, the size of an angle is determined by the internal structure and relative position of geometric shapes and is not affected by the magnification or reduction of the image.
When the image is enlarged or reduced, the size and proportion of the objects in the image will change, but this does not affect the relative position and angle between the objects. In other words, changes in the size of the image do not change the internal angular measurements of the object.
For example, if there is a triangle with a specific angle in an image, the size of the angle inside the triangle will remain the same as the image is zoomed in or out. No matter how large or small an image is, the size of the angle depends on the internal construction of the triangle and the relative positions of the sides, not on the size or proportions of the image. Since the angle remains constant as the image size changes, angle measurements in the image do not need to refer to additional parameters.

3. Code implementation

In order to facilitate understanding, the codes used in the implementation here are based on traditional digital image processing. The implementation dependency library is OpenCV and the implementation language is Python. If you want to apply it to the actual production environment, this method is not the most reliable. It is best to choose Use the deep learning-based method mentioned in the flow chart.

The implementation of the code is to identify and measure the side lengths, interior angle values, diameters, and center points of all geometric figures placed on a piece of A4 paper.

1.Data processing

First, use your mobile phone to take a few pictures of A4 paper with the geometric target you want to measure on the paper, as shown below:

Use the code to cut out the A4 area in the image: a>

from pyimagesearch import transform
from pyimagesearch import imutils
from matplotlib.patches import Polygon
import polygon_interacter as poly_i
import numpy as np
import matplotlib.pyplot as plt
import itertools
import math
import cv2
from pylsd.lsd import lsd


from scipy.spatial import distance as dist


def midpoint(ptA, ptB):
	return ((ptA[0] + ptB[0]) * 0.5, (ptA[1] + ptB[1]) * 0.5)

class DocScanner(object):
    """An image scanner"""

    def __init__(self, interactive=False, MIN_QUAD_AREA_RATIO=0.25, MAX_QUAD_ANGLE_RANGE=40):

        self.interactive = interactive
        self.MIN_QUAD_AREA_RATIO = MIN_QUAD_AREA_RATIO
        self.MAX_QUAD_ANGLE_RANGE = MAX_QUAD_ANGLE_RANGE        

    def filter_corners(self, corners, min_dist=20):
        """Filters corners that are within min_dist of others"""
        def predicate(representatives, corner):
            return all(dist.euclidean(representative, corner) >= min_dist
                       for representative in representatives)

        filtered_corners = []
        for c in corners:
            if predicate(filtered_corners, c):
                filtered_corners.append(c)
        return filtered_corners

    def angle_between_vectors_degrees(self, u, v):
        """Returns the angle between two vectors in degrees"""
        return np.degrees(
            math.acos(np.dot(u, v) / (np.linalg.norm(u) * np.linalg.norm(v))))

    def get_angle(self, p1, p2, p3):

        a = np.radians(np.array(p1))
        b = np.radians(np.array(p2))
        c = np.radians(np.array(p3))

        avec = a - b
        cvec = c - b

        return self.angle_between_vectors_degrees(avec, cvec)

    def angle_range(self, quad):

        tl, tr, br, bl = quad
        ura = self.get_angle(tl[0], tr[0], br[0])
        ula = self.get_angle(bl[0], tl[0], tr[0])
        lra = self.get_angle(tr[0], br[0], bl[0])
        lla = self.get_angle(br[0], bl[0], tl[0])

        angles = [ura, ula, lra, lla]
        return np.ptp(angles)          

    def get_corners(self, img):

        lines = lsd(img)


        corners = []
        if lines is not None:
            # separate out the horizontal and vertical lines, and draw them back onto separate canvases
            lines = lines.squeeze().astype(np.int32).tolist()
            horizontal_lines_canvas = np.zeros(img.shape, dtype=np.uint8)
            vertical_lines_canvas = np.zeros(img.shape, dtype=np.uint8)
            for line in lines:
                x1, y1, x2, y2, _ = line
                if abs(x2 - x1) > abs(y2 - y1):
                    (x1, y1), (x2, y2) = sorted(((x1, y1), (x2, y2)), key=lambda pt: pt[0])
                    cv2.line(horizontal_lines_canvas, (max(x1 - 5, 0), y1), (min(x2 + 5, img.shape[1] - 1), y2), 255, 2)
                else:
                    (x1, y1), (x2, y2) = sorted(((x1, y1), (x2, y2)), key=lambda pt: pt[1])
                    cv2.line(vertical_lines_canvas, (x1, max(y1 - 5, 0)), (x2, min(y2 + 5, img.shape[0] - 1)), 255, 2)

            lines = []

            # find the horizontal lines (connected-components -> bounding boxes -> final lines)
            (contours, hierarchy) = cv2.findContours(horizontal_lines_canvas, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_NONE)
            contours = sorted(contours, key=lambda c: cv2.arcLength(c, True), reverse=True)[:2]
            horizontal_lines_canvas = np.zeros(img.shape, dtype=np.uint8)
            for contour in contours:
                contour = contour.reshape((contour.shape[0], contour.shape[2]))
                min_x = np.amin(contour[:, 0], axis=0) + 2
                max_x = np.amax(contour[:, 0], axis=0) - 2
                left_y = int(np.average(contour[contour[:, 0] == min_x][:, 1]))
                right_y = int(np.average(contour[contour[:, 0] == max_x][:, 1]))
                lines.append((min_x, left_y, max_x, right_y))
                cv2.line(horizontal_lines_canvas, (min_x, left_y), (max_x, right_y), 1, 1)
                corners.append((min_x, left_y))
                corners.append((max_x, right_y))

            # find the vertical lines (connected-components -> bounding boxes -> final lines)
            (contours, hierarchy) = cv2.findContours(vertical_lines_canvas, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_NONE)
            contours = sorted(contours, key=lambda c: cv2.arcLength(c, True), reverse=True)[:2]
            vertical_lines_canvas = np.zeros(img.shape, dtype=np.uint8)
            for contour in contours:
                contour = contour.reshape((contour.shape[0], contour.shape[2]))
                min_y = np.amin(contour[:, 1], axis=0) + 2
                max_y = np.amax(contour[:, 1], axis=0) - 2
                top_x = int(np.average(contour[contour[:, 1] == min_y][:, 0]))
                bottom_x = int(np.average(contour[contour[:, 1] == max_y][:, 0]))
                lines.append((top_x, min_y, bottom_x, max_y))
                cv2.line(vertical_lines_canvas, (top_x, min_y), (bottom_x, max_y), 1, 1)
                corners.append((top_x, min_y))
                corners.append((bottom_x, max_y))

            # find the corners
            corners_y, corners_x = np.where(horizontal_lines_canvas + vertical_lines_canvas == 2)
            corners += zip(corners_x, corners_y)

        # remove corners in close proximity
        corners = self.filter_corners(corners)
        return corners

    def is_valid_contour(self, cnt, IM_WIDTH, IM_HEIGHT):
        """Returns True if the contour satisfies all requirements set at instantitation"""

        return (len(cnt) == 4 and cv2.contourArea(cnt) > IM_WIDTH * IM_HEIGHT * self.MIN_QUAD_AREA_RATIO 
            and self.angle_range(cnt) < self.MAX_QUAD_ANGLE_RANGE)


    def get_contour(self, rescaled_image):

        # these constants are carefully chosen
        MORPH = 9
        CANNY = 84
        HOUGH = 25

        IM_HEIGHT, IM_WIDTH, _ = rescaled_image.shape

        # convert the image to grayscale and blur it slightly
        gray = cv2.cvtColor(rescaled_image, cv2.COLOR_BGR2GRAY)
        gray = cv2.GaussianBlur(gray, (7,7), 0)

        # dilate helps to remove potential holes between edge segments
        kernel = cv2.getStructuringElement(cv2.MORPH_RECT,(MORPH,MORPH))
        dilated = cv2.morphologyEx(gray, cv2.MORPH_CLOSE, kernel)

        # find edges and mark them in the output map using the Canny algorithm
        edged = cv2.Canny(dilated, 0, CANNY)
        test_corners = self.get_corners(edged)

        approx_contours = []

        if len(test_corners) >= 4:
            quads = []

            for quad in itertools.combinations(test_corners, 4):
                points = np.array(quad)
                points = transform.order_points(points)
                points = np.array([[p] for p in points], dtype = "int32")
                quads.append(points)

            # get top five quadrilaterals by area
            quads = sorted(quads, key=cv2.contourArea, reverse=True)[:5]
            # sort candidate quadrilaterals by their angle range, which helps remove outliers
            quads = sorted(quads, key=self.angle_range)

            approx = quads[0]
            if self.is_valid_contour(approx, IM_WIDTH, IM_HEIGHT):
                approx_contours.append(approx)

        # also attempt to find contours directly from the edged image, which occasionally 
        # produces better results
        (cnts, hierarchy) = cv2.findContours(edged.copy(), cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE)
        cnts = sorted(cnts, key=cv2.contourArea, reverse=True)[:5]

        # loop over the contours
        for c in cnts:
            # approximate the contour
            approx = cv2.approxPolyDP(c, 80, True)
            if self.is_valid_contour(approx, IM_WIDTH, IM_HEIGHT):
                approx_contours.append(approx)
                break

        # If we did not find any valid contours, just use the whole image
        if not approx_contours:
            TOP_RIGHT = (IM_WIDTH, 0)
            BOTTOM_RIGHT = (IM_WIDTH, IM_HEIGHT)
            BOTTOM_LEFT = (0, IM_HEIGHT)
            TOP_LEFT = (0, 0)
            screenCnt = np.array([[TOP_RIGHT], [BOTTOM_RIGHT], [BOTTOM_LEFT], [TOP_LEFT]])

        else:
            screenCnt = max(approx_contours, key=cv2.contourArea)
            
        return screenCnt.reshape(4, 2)

    def interactive_get_contour(self, screenCnt, rescaled_image):
        poly = Polygon(screenCnt, animated=True, fill=False, color="yellow", linewidth=5)
        fig, ax = plt.subplots()
        ax.add_patch(poly)
        ax.set_title(('Drag the corners of the box to the corners of the document. \n'
            'Close the window when finished.'))
        p = poly_i.PolygonInteractor(ax, poly)
        plt.imshow(rescaled_image)
        plt.show()

        new_points = p.get_poly_points()[:4]
        new_points = np.array([[p] for p in new_points], dtype = "int32")
        return new_points.reshape(4, 2)

    def scan(self, cv_src):

        RESCALED_HEIGHT = 500.0

        ratio = cv_src.shape[0] / RESCALED_HEIGHT
        orig = cv_src.copy()
        rescaled_image = imutils.resize(cv_src, height = int(RESCALED_HEIGHT))

        # get the contour of the document
        screenCnt = self.get_contour(rescaled_image)

        if self.interactive:
            screenCnt = self.interactive_get_contour(screenCnt, rescaled_image)

        # apply the perspective transformation
        warped = transform.four_point_transform(orig, screenCnt * ratio)

        return warped


if __name__ == "__main__":
    interactive_mode = 'store_true'
    scanner = DocScanner(interactive_mode)
    cv_src = cv2.imread('1.JPG')
    cv_dst = scanner.scan(cv_src)

    cv2.namedWindow('dst',0)
    cv2.imshow('dst',cv_dst)
    cv2.waitKey()

2. Segment out the geometric figures in the image

	cv_src = cv_or.copy()

    dis_ref = dist.euclidean((start_point[0], start_point[1]), (end_point[0], end_point[1]))

    blurred_image = cv2.GaussianBlur(cv_src, (11, 11), 0)
    cv_gray = cv2.cvtColor(blurred_image, cv2.COLOR_BGR2GRAY)

    _, threshold = cv2.threshold(cv_gray, 100, 255, cv2.THRESH_BINARY)
    # threshold = cv2.adaptiveThreshold(cv_gray, 255, cv2.ADAPTIVE_THRESH_MEAN_C, cv2.THRESH_BINARY, 25,1)

    # cv2.namedWindow('th',0)
    # cv2.imshow('th',threshold)
    # cv2.waitKey()

    kernel = np.ones((13, 13), np.uint8)
    # closed_image = cv2.morphologyEx(threshold, cv2.MORPH_CLOSE, kernel)
    opened_image = cv2.morphologyEx(threshold, cv2.MORPH_OPEN, kernel)

    erosion = cv2.erode(~opened_image, (3,3), iterations=1)

    # cv2.imshow('op',erosion)

3. Identify geometric shapes

import cv2
import numpy as np

# 读取图像并转换为灰度图
image = cv2.imread('shapes.jpg')
gray = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)

# 阈值化图像
_, thresh = cv2.threshold(gray, 240, 255, cv2.THRESH_BINARY)

# 查找轮廓
contours, _ = cv2.findContours(thresh, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE)

for contour in contours:
    # 计算轮廓的逼近多边形
    approx = cv2.approxPolyDP(contour, 0.04 * cv2.arcLength(contour, True), True)
    
    # 区分形状
    if len(approx) == 3:
        shape = "Triangle"
    elif len(approx) == 4:
        shape = "Rectangle"
    else:
        shape = "Circle"

    # 在图像上绘制轮廓和形状名称
    cv2.drawContours(image, [contour], -1, (0, 255, 0), 2)
    cv2.putText(image, shape, (contour[0][0][0], contour[0][0][1]), cv2.FONT_HERSHEY_SIMPLEX, 
                0.5, (255, 255, 255), 2)

# 显示结果图像
cv2.imshow("Shapes", image)
cv2.waitKey(0)
cv2.destroyAllWindows()

4. Line segment length measurement

def measure_length(line,width,dis_ref):
    distance = dist.euclidean((line[0][0],line[0][1]), (line[1][0],line[1][1]))
    pixelsPerMetric = dis_ref / width
    dim = distance / pixelsPerMetric

    midpoint = ((line[0][0] + line[1][0]) // 2, (line[0][1] + line[1][1]) // 2)

    return dim,midpoint

5.Angle measurement

#获取两条线的角度，返回角度值与交点
def measure_angle(line1,line2):
    slope1 = (line1[1][1] - line1[0][1]) / (line1[1][0] - line1[0][0])  # 斜率1
    slope2 = (line2[1][1] - line2[0][1]) / (line2[1][0] - line2[0][0])  # 斜率2

    # 计算交点
    x_intersect = (slope1 * line1[0][0] - slope2 * line2[0][0] + line2[0][1] - line1[0][1]) / (
            slope1 - slope2)
    y_intersect = slope1 * (x_intersect - line1[0][0]) + line1[0][1]

    # 计算两条线之间的角度（弧度）
    angle_rad = np.arctan(abs((slope2 - slope1) / (1 + slope1 * slope2)))

    # 将弧度转换为角度
    angle_deg = np.degrees(angle_rad)

    angle = round(angle_deg, 1)

    return angle,(int(x_intersect), int(y_intersect))

Test results: