Steger algorithm implements structured light strip center extraction (python version)

Steger algorithm principle

When extracting the light strip center of structured light, the Steger algorithm is based on the Hessian matrix. Its basic steps are as follows:

Find the normal direction of the line laser stripe from the Hessian matrix
Expand its grayscale distribution according to Taylor polynomials in the normal direction of the light stripe, and the maximum value obtained is the sub-pixel coordinate of the light stripe in the normal direction. For a two-dimensional discrete image $I(u,v)$ , the Hessian matrix can be expressed as:

$H(u,v) = \begin{bmatrix} I_{uu} &I_{uv} \\ I_{uv} & I_{vv} \end{bmatrix}$

The u and v here represent the row and column coordinates of the pixel. $I_{uv}$ represents the grayscale of the pixel. $(u,v)$ can also be called Grayscale distribution function. And $I_{uu}$ , $I_{uv}$ and $I_{vv}$ can be calculated by $I_{uv}$ and the two-dimensional Gaussian function $G(u,v)$ Obtained by convolution operation.

$G(u,v) = \frac{1}{\sigma\sqrt{2 \pi}}\cdot e^{\left ( -\frac{u^{2}+v^{2}}{2\sigma^{2}} \right )}$

$I_{uu}=\frac{\partial ^{2}}{\partial u\partial u}G(u,v)\bigotimes I_{uv}$

$I_{uv}=\frac{\partial ^{2}}{\partial u\partial v}G(u,v)\bigotimes I_{uv}$

$I_{vv}=\frac{\partial ^{2}}{\partial v\partial v}G(u,v)\bigotimes I_{uv}$

Here, the two-dimensional Gaussian function is used to make the grayscale distribution characteristics of the light strip more obvious. In the G(u,v) expression, $\sigma$ is the standard The difference is generally taken as $\sigma \geq \frac{W}{\sqrt{3}}$ , and W represents the width of the light bar. The Hessian matrix at the pixel (u, v) has two eigenvectors, one of which has a larger absolute value and is the normal direction vector at the pixel, while the other is the tangential direction vector. Therefore, the normal direction can be calculated by obtaining the eigenvector of the Hessian matrix. For a certain pixel $H(u_{0},v_{0})$ , the second-order Taylor expansion Hessian matrix is:

$H(u_{0},v_{0}) = \begin{bmatrix} I_{uu} &I_{uv} \\ I_{uv} & I_{vv} \end{bmatrix}$

The eigenvalues and eigenvectors calculated from the Hessian matrix of the point correspond to the normal direction of the point and the second-order directional derivative of the direction respectively. The unit vector in the normal direction is: $e = [e_{u},e_{v}]$ , and the pixel point in the normal direction of the light stripe $(u_{0}+t\cdot e_{u},v_{0}+t\cdot e_{v})$ , and the pixel in the normal direction of the light stripe Point $I = (u_{0}+t\cdot e_{u},v_{0}+t\cdot e_{v})$ can be expressed by the gray level I(u0,v0) of pixel (u0,v0) and the second-order Taylor expansion polynomial as:

$I (u_{0}+t\cdot e_{u},v_{0}+t\cdot e_{v}) = I(u_{0},v_{0})+t\cdot e[I_{u},I_{v}]^{T} +t\cdot e\cdot H(u,v) \cdot e^{T}$

$t = -\frac{e_{u}\cdot I_{u}+e_{v}\cdot I_{v}}{e_{u}^{2}\cdot I_{uu}+2\cdot e_{u}\cdot e_{v}\cdot I_{uv}+e_{v}^{2}\cdot I_{vv}}$

Then substitute t (i.e. Taylor expansion) into it to obtain the sub-pixel coordinates of the light stripe center.

Steger algorithm Python implementation

In this part, I found many C++ version of the Steger algorithm implementation on the Internet. I made reference to the existing C++ version of the Steger algorithm implementation, made improvements on this basis, and re-implemented the algorithm in Python.

Calculate the first and second derivatives of an image

Here I used three methods, namely custom convolution, Scharr filter, and Sobel filter to calculate the derivative and second-order derivative of the image.

import cv2
import numpy as np

def _derivation_with_Filter(Gaussimg):
    dx = cv2.filter2D(Gaussimg,-1, kernel=np.array([[1], [0], [-1]]))
    dy = cv2.filter2D(Gaussimg,-1, kernel=np.array([[1, 0, -1]]))
    dxx = cv2.filter2D(Gaussimg,-1, kernel=np.array([[1], [-2], [1]]))
    dyy = cv2.filter2D(Gaussimg,-1, kernel=np.array([[1, -2, 1]]))
    dxy = cv2.filter2D(Gaussimg,-1, kernel=np.array([[1, -1], [-1, 1]]))

    return dx, dy, dxx, dyy, dxy

def _derivation_with_Scharr(Gaussimg):
    dx = cv2.Scharr(Gaussimg, cv2.CV_32F, 1, 0)
    dy = cv2.Scharr(Gaussimg, cv2.CV_32F, 0, 1)
    dxx = cv2.Scharr(dx, cv2.CV_32F, 1, 0)
    dxy = cv2.Scharr(dx, cv2.CV_32F, 0, 1)
    dyy = cv2.Scharr(dy, cv2.CV_32F, 0, 1)

    return dx, dy, dxx, dyy, dxy

def _derivation_with_Sobel(Gaussimg):
    dx = cv2.Sobel(Gaussimg, cv2.CV_32F, 1, 0, ksize=3)
    dy = cv2.Sobel(Gaussimg, cv2.CV_32F, 0, 1, ksize=3)
    dxx = cv2.Sobel(dx, cv2.CV_32F, 1, 0, ksize=3)
    dxy = cv2.Sobel(dx, cv2.CV_32F, 0, 1, ksize=3)
    dyy = cv2.Sobel(dy, cv2.CV_32F, 0, 1, ksize=3)

    return dx, dy, dxx, dyy, dxy

if __name__=="__main__":
    from pyzjr.dlearn import Runcodes
    sigmaX, sigmaY = 1.1, 1.1

    img = cv2.imread(r"D:\PythonProject\net\line\linedata\Image_1.jpg")
    gray_img = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)
    Gaussimg = cv2.GaussianBlur(gray_img, ksize=(0, 0), sigmaX=sigmaX, sigmaY=sigmaY)
    with Runcodes("Filter"):
        dx, dy, dxx, dyy, dxy = _derivation_with_Filter(Gaussimg)
        print(type(dx[0][0]))
    with Runcodes("Scharr"):
        dx1, dy1, dxx1, dyy1, dxy1 = _derivation_with_Scharr(Gaussimg)
        print(type(dx1[0][0]))
    with Runcodes("Sobel"):
        dx2, dy2, dxx2, dyy2, dxy2 = _derivation_with_Sobel(Gaussimg)
        print(type(dx2[0][0]))

The output in the console is:

<class 'numpy.uint8'>
Filter: 0.00602 sec
<class 'numpy.float32'>
Scharr: 0.00770 sec
<class 'numpy.float32'>
Sobel: 0.01025 sec

In terms of operation time, the custom Filter is the fastest, the scharr filter is in the middle, and the sobel filter is the slowest; in terms of element types, the first one is uint8, and the other two are float32 types.

The Hessian matrix can be created by selecting a calculation method.

def Magnitudefield(dxx, dyy):
    """计算幅度和相位"""
    dxx = dxx.astype(float)
    dyy = dyy.astype(float)
    mag = cv2.magnitude(dxx, dyy)
    phase = mag*180./np.pi
    return mag, phase

def derivation(gray_img, sigmaX, sigmaY, method="Scharr", nonmax=False):
    """
    计算图像的一阶导数 dx 和 dy，以及二阶导数 dxx、dyy 和 dxy
    :param gray_img: 灰度图
    :param sigmaX: 在水平方向的高斯核标准差,用于激光线提取建议取1-2
    :param sigmaY: 在垂直方向上的高斯核标准差,用于激光线提取建议取1-2
    :param method:"Scharr"  or  "Filter"  or  "Sobel"
                  选择什么方式获取dx, dy, dxx, dyy, dxy,提供了卷积与Scharr和Sobel滤波器三种方式计算,
                  Scharr滤波器通常会产生更平滑和准确的结果,所以这里默认使用"Scharr"方法,虽然
                  "Sobel"运行比另外两种要慢,但在使用的时候,建议自己试试
    :return: dx, dy, dxx, dyy, dxy
    """
    Gaussimg = cv2.GaussianBlur(gray_img, ksize=(0, 0), sigmaX=sigmaX, sigmaY=sigmaY)
    dx, dy, dxx, dyy, dxy = _derivation_with_Scharr(Gaussimg)
    if method == "Filter":
        dx, dy, dxx, dyy, dxy = _derivation_with_Filter(Gaussimg)
    elif method == "Sobel":
        dx, dy, dxx, dyy, dxy =_derivation_with_Sobel(Gaussimg)
    if nonmax:
        normal, phase = Magnitudefield(dxx, dyy)
        dxy = nonMaxSuppression(normal, phase)
    return dx, dy, dxx, dyy, dxy

def nonMaxSuppression(det, phase):
    """非最大值抑制"""
    gmax = np.zeros(det.shape)
    # thin-out evry edge for angle = [0, 45, 90, 135]
    for i in range(gmax.shape[0]):
        for j in range(gmax.shape[1]):
            if phase[i][j] < 0:
                phase[i][j] += 360
            if ((j+1) < gmax.shape[1]) and ((j-1) >= 0) and ((i+1) < gmax.shape[0]) and ((i-1) >= 0):
                # 0 degrees
                if (phase[i][j] >= 337.5 or phase[i][j] < 22.5) or (phase[i][j] >= 157.5 and phase[i][j] < 202.5):
                    if det[i][j] >= det[i][j + 1] and det[i][j] >= det[i][j - 1]:
                        gmax[i][j] = det[i][j]
                # 45 degrees
                if (phase[i][j] >= 22.5 and phase[i][j] < 67.5) or (phase[i][j] >= 202.5 and phase[i][j] < 247.5):
                    if det[i][j] >= det[i - 1][j + 1] and det[i][j] >= det[i + 1][j - 1]:
                        gmax[i][j] = det[i][j]
                # 90 degrees
                if (phase[i][j] >= 67.5 and phase[i][j] < 112.5) or (phase[i][j] >= 247.5 and phase[i][j] < 292.5):
                    if det[i][j] >= det[i - 1][j] and det[i][j] >= det[i + 1][j]:
                        gmax[i][j] = det[i][j]
                # 135 degrees
                if (phase[i][j] >= 112.5 and phase[i][j] < 157.5) or (phase[i][j] >= 292.5 and phase[i][j] < 337.5):
                    if det[i][j] >= det[i - 1][j - 1] and det[i][j] >= det[i + 1][j + 1]:
                        gmax[i][j] = det[i][j]
    return gmax

The Magnitudefield(dxx, dyy) function calculates the magnitude and phase of the gradient, using the given x and y derivatives (dxx and dyy). The amplitude represents the strength of the gradient, and the phase represents the direction of the gradient; the nonMaxSuppression(det, phase) function performs non-maximum suppression on the gradient amplitude (det) according to the phase of the gradient (phase). It helps to retain only local maxima in the gradient direction, thereby refining the detected edges; it is then applied to the derivation function for regulation.

Second-order Taylor expansion Hessian matrix

def HessianMatrix(self, dx, dy, dxx, dyy, dxy, threshold=0.5):
    """
    HessianMatrix = [dxx    dxy]
                    [dxy    dyy]
    compute hessian:
            [dxx   dxy]         [00    01]
                         ====>
            [dxy   dyy]         [10    11]
    """
    point=[]
    direction=[]
    value=[]
    for x in range(0, self.col):
        for y in range(0, self.row):
            if dxy[y,x] > 0:
                hessian = np.zeros((2,2))
                hessian[0,0] = dxx[y,x]
                hessian[0,1] = dxy[y,x]
                hessian[1,0] = dxy[y,x]
                hessian[1,1] = dyy[y,x]
                # 计算矩阵的特征值和特征向量
                _, eigenvalues, eigenvectors = cv2.eigen(hessian)
                if np.abs(eigenvalues[0,0]) >= np.abs(eigenvalues[1,0]):
                    nx = eigenvectors[0,0]
                    ny = eigenvectors[0,1]
                else:
                    nx = eigenvectors[1,0]
                    ny = eigenvectors[1,1]

                # Taylor展开式分子分母部分,需要避免为0的情况
                Taylor_numer = (dx[y, x] * nx + dy[y, x] * ny)
                Taylor_denom = dxx[y,x]*nx*nx + dyy[y,x]*ny*ny + 2*dxy[y,x]*nx*ny
                if Taylor_denom != 0:
                    T = -(Taylor_numer/Taylor_denom)
                    # Hessian矩阵最大特征值对应的特征向量对应于光条的法线方向
                    if np.abs(T*nx) <= threshold and np.abs(T*ny) <= threshold:
                        point.append((x,y))
                        direction.append((nx,ny))
                        value.append(np.abs(dxy[y,x]+dxy[y,x]))
    return point, direction, value

I defined this part under the steger class

Draw the center line in the background and original image

def centerline(self,img, sigmaX, sigmaY, method="Scharr", usenonmax=True, color=(0, 0, 255)):
    dx, dy, dxx, dyy, dxy = derivation(self.gray_image, sigmaX, sigmaY, method=method, nonmax=usenonmax)
    point, direction, value = self.HessianMatrix(dx, dy, dxx, dyy, dxy)
    for point in point:
        self.newimage[point[1], point[0]] = 255
        img[point[1], point[0], :] = color

    return img, self.newimage

Code and effect display

import cv2
import numpy as np

def _derivation_with_Filter(Gaussimg):
    dx = cv2.filter2D(Gaussimg,-1, kernel=np.array([[1], [0], [-1]]))
    dy = cv2.filter2D(Gaussimg,-1, kernel=np.array([[1, 0, -1]]))
    dxx = cv2.filter2D(Gaussimg,-1, kernel=np.array([[1], [-2], [1]]))
    dyy = cv2.filter2D(Gaussimg,-1, kernel=np.array([[1, -2, 1]]))
    dxy = cv2.filter2D(Gaussimg,-1, kernel=np.array([[1, -1], [-1, 1]]))

    return dx, dy, dxx, dyy, dxy

def _derivation_with_Scharr(Gaussimg):
    dx = cv2.Scharr(Gaussimg, cv2.CV_32F, 1, 0)
    dy = cv2.Scharr(Gaussimg, cv2.CV_32F, 0, 1)
    dxx = cv2.Scharr(dx, cv2.CV_32F, 1, 0)
    dxy = cv2.Scharr(dx, cv2.CV_32F, 0, 1)
    dyy = cv2.Scharr(dy, cv2.CV_32F, 0, 1)

    return dx, dy, dxx, dyy, dxy

def _derivation_with_Sobel(Gaussimg):
    dx = cv2.Sobel(Gaussimg, cv2.CV_32F, 1, 0, ksize=3)
    dy = cv2.Sobel(Gaussimg, cv2.CV_32F, 0, 1, ksize=3)
    dxx = cv2.Sobel(dx, cv2.CV_32F, 1, 0, ksize=3)
    dxy = cv2.Sobel(dx, cv2.CV_32F, 0, 1, ksize=3)
    dyy = cv2.Sobel(dy, cv2.CV_32F, 0, 1, ksize=3)

    return dx, dy, dxx, dyy, dxy

def Magnitudefield(dxx, dyy):
    """计算幅度和相位"""
    dxx = dxx.astype(float)
    dyy = dyy.astype(float)
    mag = cv2.magnitude(dxx, dyy)
    phase = mag*180./np.pi
    return mag, phase

def derivation(gray_img, sigmaX, sigmaY, method="Scharr", nonmax=False):
    """
    计算图像的一阶导数 dx 和 dy，以及二阶导数 dxx、dyy 和 dxy
    :param gray_img: 灰度图
    :param sigmaX: 在水平方向的高斯核标准差,用于激光线提取建议取1-2
    :param sigmaY: 在垂直方向上的高斯核标准差,用于激光线提取建议取1-2
    :param method:"Scharr"  or  "Filter"  or  "Sobel"
                  选择什么方式获取dx, dy, dxx, dyy, dxy,提供了卷积与Scharr和Sobel滤波器三种方式计算,
                  Scharr滤波器通常会产生更平滑和准确的结果,所以这里默认使用"Scharr"方法,虽然
                  "Sobel"运行比另外两种要慢,但在使用的时候,建议自己试试
    :return: dx, dy, dxx, dyy, dxy
    """
    Gaussimg = cv2.GaussianBlur(gray_img, ksize=(0, 0), sigmaX=sigmaX, sigmaY=sigmaY)
    dx, dy, dxx, dyy, dxy = _derivation_with_Scharr(Gaussimg)
    if method == "Filter":
        dx, dy, dxx, dyy, dxy = _derivation_with_Filter(Gaussimg)
    elif method == "Sobel":
        dx, dy, dxx, dyy, dxy =_derivation_with_Sobel(Gaussimg)
    if nonmax:
        normal, phase = Magnitudefield(dxx, dyy)
        dxy = nonMaxSuppression(normal, phase)
    return dx, dy, dxx, dyy, dxy

def nonMaxSuppression(det, phase):
    """非最大值抑制"""
    gmax = np.zeros(det.shape)
    # thin-out evry edge for angle = [0, 45, 90, 135]
    for i in range(gmax.shape[0]):
        for j in range(gmax.shape[1]):
            if phase[i][j] < 0:
                phase[i][j] += 360
            if ((j+1) < gmax.shape[1]) and ((j-1) >= 0) and ((i+1) < gmax.shape[0]) and ((i-1) >= 0):
                # 0 degrees
                if (phase[i][j] >= 337.5 or phase[i][j] < 22.5) or (phase[i][j] >= 157.5 and phase[i][j] < 202.5):
                    if det[i][j] >= det[i][j + 1] and det[i][j] >= det[i][j - 1]:
                        gmax[i][j] = det[i][j]
                # 45 degrees
                if (phase[i][j] >= 22.5 and phase[i][j] < 67.5) or (phase[i][j] >= 202.5 and phase[i][j] < 247.5):
                    if det[i][j] >= det[i - 1][j + 1] and det[i][j] >= det[i + 1][j - 1]:
                        gmax[i][j] = det[i][j]
                # 90 degrees
                if (phase[i][j] >= 67.5 and phase[i][j] < 112.5) or (phase[i][j] >= 247.5 and phase[i][j] < 292.5):
                    if det[i][j] >= det[i - 1][j] and det[i][j] >= det[i + 1][j]:
                        gmax[i][j] = det[i][j]
                # 135 degrees
                if (phase[i][j] >= 112.5 and phase[i][j] < 157.5) or (phase[i][j] >= 292.5 and phase[i][j] < 337.5):
                    if det[i][j] >= det[i - 1][j - 1] and det[i][j] >= det[i + 1][j + 1]:
                        gmax[i][j] = det[i][j]
    return gmax


class Steger():
    def __init__(self, gray_image):
        self.gray_image = gray_image
        self.row, self.col = self.gray_image.shape[:2]
        self.newimage = np.zeros((self.row, self.col), np.uint8)

    def centerline(self,img, sigmaX, sigmaY, method="Scharr", usenonmax=True, color=(0, 0, 255)):
        dx, dy, dxx, dyy, dxy = derivation(self.gray_image, sigmaX, sigmaY, method=method, nonmax=usenonmax)
        point, direction, value = self.HessianMatrix(dx, dy, dxx, dyy, dxy)
        for point in point:
            self.newimage[point[1], point[0]] = 255
            img[point[1], point[0], :] = color
    
        return img, self.newimage

    def HessianMatrix(self, dx, dy, dxx, dyy, dxy, threshold=0.5):
        """
        HessianMatrix = [dxx    dxy]
                        [dxy    dyy]
        compute hessian:
                [dxx   dxy]         [00    01]
                             ====>
                [dxy   dyy]         [10    11]
        """
        point=[]
        direction=[]
        value=[]
        for x in range(0, self.col):
            for y in range(0, self.row):
                if dxy[y,x] > 0:
                    hessian = np.zeros((2,2))
                    hessian[0,0] = dxx[y,x]
                    hessian[0,1] = dxy[y,x]
                    hessian[1,0] = dxy[y,x]
                    hessian[1,1] = dyy[y,x]
                    # 计算矩阵的特征值和特征向量
                    _, eigenvalues, eigenvectors = cv2.eigen(hessian)
                    if np.abs(eigenvalues[0,0]) >= np.abs(eigenvalues[1,0]):
                        nx = eigenvectors[0,0]
                        ny = eigenvectors[0,1]
                    else:
                        nx = eigenvectors[1,0]
                        ny = eigenvectors[1,1]

                    # Taylor展开式分子分母部分,需要避免为0的情况
                    Taylor_numer = (dx[y, x] * nx + dy[y, x] * ny)
                    Taylor_denom = dxx[y,x]*nx*nx + dyy[y,x]*ny*ny + 2*dxy[y,x]*nx*ny
                    if Taylor_denom != 0:
                        T = -(Taylor_numer/Taylor_denom)
                        # Hessian矩阵最大特征值对应的特征向量对应于光条的法线方向
                        if np.abs(T*nx) <= threshold and np.abs(T*ny) <= threshold:
                            point.append((x,y))
                            direction.append((nx,ny))
                            value.append(np.abs(dxy[y,x]+dxy[y,x]))
        return point, direction, value


def threshold(mask, std=127.5):
    """阈值法"""
    mask[mask > std] = 255
    mask[mask < std] = 0
    return mask.astype("uint8")

def Bessel(xi: list):
    """贝塞尔公式"""
    xi_array = np.array(xi)
    x_average = np.mean(xi_array)
    squared_diff = (xi_array - x_average) ** 2
    variance = squared_diff / (len(xi)-1)
    bessel = np.sqrt(variance)

    return bessel

def test_Steger():
    img = cv2.imread(r"D:\PythonProject\net\line\data\Image_20230215160728679.jpg")

    gray_img = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)
    gray_img = threshold(gray_img)
    steger = Steger(gray_img)
    img,newimage=steger.centerline(img,sigmaX=1.1,sigmaY=1.1,method="Sobel")
    # point, direction, value = HessianMatrix(gray_img, 1.1, 1.1, usenonmax=True)


    cv2.imwrite("result/main3.png", newimage)
    cv2.imwrite("result/main3_img.png", img)



def test_derivation_methods_time():
    from pyzjr.dlearn import Runcodes
    img = cv2.imread(r"D:\PythonProject\net\line\data\Image_20230215160728411.jpg")
    img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)
    gray_img = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)
    with Runcodes(description="Filter"):
        print(derivation(gray_img,1.1,1.1,"Filter"))
        # Filter: 0.01344 sec

    with Runcodes(description="Scharr"):
        print(derivation(gray_img, 1.1, 1.1))
        # Scharr: 0.00959 sec

    with Runcodes(description="Sobel"):
        print(derivation(gray_img, 1.1, 1.1,"Sobel"))
        # Sobel: 0.01820 sec



if __name__=="__main__":
    # test_derivation_methods_time()
    test_Steger()

I still haven’t figured out why there are three lines. It seems to be the center line and the outline. I will update it again if there is any improvement.