Step by step learning OAK six: feature detection through OAK camera

feature detection

Feature detection refers to automatically finding and locating image regions or image points with specific features in digital images. These features can be objects, edges, corners, textures, etc. that have unique appearance, structure, or statistical properties in the image.

Feature detection plays a vital role in computer vision, and it is the basis of many computer vision tasks, such as object detection, tracking, pose estimation, image stitching, image recognition, etc. By extracting and matching features, computer vision tasks such as image alignment, object recognition, and motion estimation can be achieved.

Traditional feature detection methods include edge detection (such as Canny edge detection), corner detection (such as Harris corner detection), scale invariant feature transform (SIFT), accelerated robust feature (SURF), etc. In recent years, feature detection methods based on deep learning, such as convolutional neural network (CNN) feature extraction, have also made important breakthroughs and applications in the field of image feature detection.

Setup 1: Create the file

• Create a new 6-feature-detector folder
• Open the folder with vscode
• Create a new main.py file

Setup 2: Install dependencies

Before installing dependencies, you need to create and activate a virtual environment. I have created a virtual environment OAKenv here, enter cd in the terminal... to return to the root directory of OAKenv, and enter to activate the virtual OAKenv\Scripts\activateenvironment

Install pip dependencies:

pip install numpy opencv-python depthai blobconverter --user

Setup 3: Import required packages

Import the packages required by the project in main.py

import cv2
import depthai as dai

Setup 4: Create the pipeline

pipeline = dai.Pipeline()

Setup 5: Create Nodes

Create a camera node

monoLeft = pipeline.createMonoCamera()
monoRight = pipeline.createMonoCamera()

1. monoLeft = pipeline.createMonoCamera(): Create a single-channel camera node (MonoCamera) for capturing the left monochrome image.
2. monoRight = pipeline.createMonoCamera(): Create a single-channel camera node (MonoCamera) for capturing the monochrome image on the right.

Create a feature detection node

featureTrackerLeft = pipeline.createFeatureTracker()
featureTrackerRight = pipeline.createFeatureTracker()

This code creates three edge detection nodes in the pipeline.

1. featureTrackerLeft = pipeline.createFeatureTracker(): Created a feature detection node to process the left monochrome image.
2. featureTrackerRight = pipeline.createFeatureTracker(): Created a feature detection node to process the monochrome image on the right.

Create a node for data interaction

xoutPassthroughFrameLeft = pipeline.createXLinkOut()
xoutTrackedFeaturesLeft = pipeline.createXLinkOut()
xoutPassthroughFrameRight = pipeline.createXLinkOut()
xoutTrackedFeaturesRight = pipeline.createXLinkOut()
xinTrackedFeaturesConfig = pipeline.createXLinkIn()

xoutPassthroughFrameLeft.setStreamName("passthroughFrameLeft")
xoutTrackedFeaturesLeft.setStreamName("trackedFeaturesLeft")
xoutPassthroughFrameRight.setStreamName("passthroughFrameRight")
xoutTrackedFeaturesRight.setStreamName("trackedFeaturesRight")
xinTrackedFeaturesConfig.setStreamName("trackedFeaturesConfig")

This code creates five nodes for data interaction with external devices.

1. Using pipeline.createXLinkOut()the method creates four nodes for data output with external devices.
2. The usage pipeline.createXLinkIn()method creates a node for data input with an external device.

A dataflow name is set for each output node and input node

Use .setStreamName("name")the method to set the dataflow name for the five nodes created above

Setup 6: Set related properties

Set the relevant properties of the camera

monoLeft.setResolution(dai.MonoCameraProperties.SensorResolution.THE_400_P)
monoLeft.setBoardSocket(dai.CameraBoardSocket.LEFT)
monoRight.setResolution(dai.MonoCameraProperties.SensorResolution.THE_400_P)
monoRight.setBoardSocket(dai.CameraBoardSocket.RIGHT)

For the monocular camera on the left:

1. monoLeft.setResolution(dai.MonoCameraProperties.SensorResolution.THE_400_P): Set the resolution of the left monocular camera to 400P. This means that the left monocular camera will capture images at 400P resolution.
2. monoLeft.setBoardSocket(dai.CameraBoardSocket.LEFT): Set the onboard slot of the left monocular camera to LEFT. This indicates that the left monocular camera will be connected via the left connector on the system board.

For the monocular camera on the right:

1. monoRight.setResolution(dai.MonoCameraProperties.SensorResolution.THE_400_P): Set the resolution of the right monocular camera to 400P. This means that the right monocular camera will capture images at 400P resolution.
2. monoRight.setBoardSocket(dai.CameraBoardSocket.RIGHT): Set the onboard slot of the right monocular camera to RIGHT. This indicates that the right monocular camera will be connected via the right connector on the system board.

Set up the initial configuration of the feature detector

featureTrackerLeft.initialConfig.setMotionEstimator(False)
featureTrackerRight.initialConfig.setMotionEstimator(False)

Disables the motion estimator in a feature tracker instance.

For each feature tracker instance, initialConfigis its initialization configuration object, which can be used to set different parameters.

Disable the motion estimator by calling setMotionEstimator(False)the function and passing it as an argument. FalseThis means that the feature tracker will not estimate the motion between objects or feature points, but only track their positions.

Setup 7: Establish link relationship

Establish data connection between camera and feature tracker

monoLeft.out.link(featureTrackerLeft.inputImage)
featureTrackerLeft.passthroughInputImage.link(xoutPassthroughFrameLeft.input)
featureTrackerLeft.outputFeatures.link(xoutTrackedFeaturesLeft.input)
xinTrackedFeaturesConfig.out.link(featureTrackerLeft.inputConfig)

monoRight.out.link(featureTrackerRight.inputImage)
featureTrackerRight.passthroughInputImage.link(xoutPassthroughFrameRight.input)
featureTrackerRight.outputFeatures.link(xoutTrackedFeaturesRight.input)
xinTrackedFeaturesConfig.out.link(featureTrackerRight.inputConfig)

featureTrackerConfig = featureTrackerRight.initialConfig.get()

print("Press 's' to switch between Harris and Shi-Thomasi corner detector!")

1. monoLeft.out.link(featureTrackerLeft.inputImage): Link the output of the left monocular camera to the input image of the left feature tracker.
2. featureTrackerLeft.passthroughInputImage.link(xoutPassthroughFrameLeft.input): Links the input image of the left feature tracker to the left frame output (used to pass the original input image to the output pipeline).
3. featureTrackerLeft.outputFeatures.link(xoutTrackedFeaturesLeft.input): Links the output feature of the left feature tracker to the output of the left tracked feature.
4. xinTrackedFeaturesConfig.out.link(featureTrackerLeft.inputConfig): Links the configuration of the tracked feature to the input configuration of the left feature tracker.
5. monoRight.out.link(featureTrackerRight.inputImage): Link the output of the right monocular camera to the input image of the right feature tracker.
6. featureTrackerRight.passthroughInputImage.link(xoutPassthroughFrameRight.input): Links the input image of the right feature tracker to the right frame output.
7. featureTrackerRight.outputFeatures.link(xoutTrackedFeaturesRight.input): Links the output feature of the right feature tracker to the output of the right tracked feature.
8. xinTrackedFeaturesConfig.out.link(featureTrackerRight.inputConfig): Links the configuration of the tracked feature to the input configuration of the feature tracker on the right.
9. featureTrackerRight.initialConfig.get(): Used to get featureTrackerRightthe initial configuration of the feature tracker instance.

This code snippet is mainly used to create a data pipeline that passes image data from the camera to the feature tracker and outputs the tracked features to the corresponding targets.

Setup 8: Connect the device and start the pipeline

with dai.Device(pipeline) as device:

Setup 9: Create input queue and output queue to communicate with DepthAI device

    passthroughImageLeftQueue = device.getOutputQueue("passthroughFrameLeft", 8, False)
    outputFeaturesLeftQueue = device.getOutputQueue("trackedFeaturesLeft", 8, False)
    passthroughImageRightQueue = device.getOutputQueue("passthroughFrameRight", 8, False)
    outputFeaturesRightQueue = device.getOutputQueue("trackedFeaturesRight", 8, False)

    inputFeatureTrackerConfigQueue = device.getInputQueue("trackedFeaturesConfig")
    
    leftWindowName = "left"
    rightWindowName = "right"

1. passthroughImageLeftQueue = device.getOutputQueue("passthroughFrameLeft", 8, False): Get the output queue named "passthroughFrameLeft" from the device. This queue will be used to receive the frame image data of the left feature tracker, the queue size is 8, and no blocking mode is used.
2. outputFeaturesLeftQueue = device.getOutputQueue("trackedFeaturesLeft", 8, False): Get an output queue named "trackedFeaturesLeft" from the device. This queue will be used to receive the tracking feature data of the left feature tracker, the queue size is 8, and the blocking mode is not used.
3. passthroughImageRightQueue = device.getOutputQueue("passthroughFrameRight", 8, False): Get the output queue named "passthroughFrameRight" from the device. This queue will be used to receive the frame image data of the feature tracker on the right, the queue size is 8, and no blocking mode is used.
4. outputFeaturesRightQueue = device.getOutputQueue("trackedFeaturesRight", 8, False): Get an output queue named "trackedFeaturesRight" from the device. This queue will be used to receive the tracking feature data of the feature tracker on the right, the queue size is 8, and the blocking mode is not used.
5. inputFeatureTrackerConfigQueue = device.getInputQueue("trackedFeaturesConfig"): Get an input queue named "trackedFeaturesConfig" from the device. This queue will be used to send configuration data to the feature tracker.

Setup 10: Defined drawFeaturesfunctions for drawing feature points on the image frame

    def drawFeatures(frame, features):
        pointColor = (0, 0, 255)
        circleRadius = 2
        for feature in features:
            cv2.circle(frame, (int(feature.position.x), int(feature.position.y)), circleRadius, pointColor, -1, cv2.LINE_AA, 0)

defines a drawFeaturesfunction named , which is used to draw feature points on an image frame.

1. pointColor = (0, 0, 255): Defines the color used when drawing feature points. Here red in the RGB color space is used.

2. circleRadius = 2: Defines the radius of the circle used when drawing feature points.

3. for feature in features:: Loop through each feature in the feature list.

4. cv2.circle(frame, (int(feature.position.x), int(feature.position.y)), circleRadius, pointColor, -1, cv2.LINE_AA, 0): Use the function of OpenCV to draw a circle on cv2.circlethe image frame . frameThe position of the center of the circle is determined by the position of the feature (feature.position.x, feature.position.y), the radius of the circle circleRadiusis , the color is pointColor, the line width is -1 to fill the entire circle, cv2.LINE_AAspecifying the use of anti-aliased line segments, and the last parameter is 0 to draw in the default image coordinate system.

Setup 11: Main loop

    while True:

Get frame data and convert it to color

        inPassthroughFrameLeft = passthroughImageLeftQueue.get()
        passthroughFrameLeft = inPassthroughFrameLeft.getFrame()
        leftFrame = cv2.cvtColor(passthroughFrameLeft, cv2.COLOR_GRAY2BGR)

        inPassthroughFrameRight = passthroughImageRightQueue.get()
        passthroughFrameRight = inPassthroughFrameRight.getFrame()
        rightFrame = cv2.cvtColor(passthroughFrameRight, cv2.COLOR_GRAY2BGR)

1. inPassthroughFrameLeft = passthroughImageLeftQueue.get()passthroughImageLeftQueue: Get a frame from the named queue.

2. passthroughFrameLeft = inPassthroughFrameLeft.getFrame()inPassthroughFrameLeft: Extract frame data from the obtained frame object .

3. leftFrame = cv2.cvtColor(passthroughFrameLeft, cv2.COLOR_GRAY2BGR)cv2.cvtColor: Convert a grayscale image passthroughFrameLeftto a color image using OpenCV's functions. The color conversion code used here cv2.COLOR_GRAY2BGRconverts a grayscale image to a BGR color image.

4. inPassthroughFrameRight = passthroughImageRightQueue.get()passthroughImageRightQueue: Get another frame from the named queue.

5. passthroughFrameRight = inPassthroughFrameRight.getFrame()inPassthroughFrameRight: Extract frame data from the obtained frame object .

6. rightFrame = cv2.cvtColor(passthroughFrameRight, cv2.COLOR_GRAY2BGR)cv2.cvtColor: Convert a grayscale image passthroughFrameRightto a color image using OpenCV's functions. The color conversion code used here cv2.COLOR_GRAY2BGRconverts a grayscale image to a BGR color image.

Get the tracked features and draw those features on the frame

        trackedFeaturesLeft = outputFeaturesLeftQueue.get().trackedFeatures
        drawFeatures(leftFrame, trackedFeaturesLeft)

        trackedFeaturesRight = outputFeaturesRightQueue.get().trackedFeatures
        drawFeatures(rightFrame, trackedFeaturesRight)

This code takes the tracked features from the output feature queue and draws them on the frame.

1. trackedFeaturesLeft = outputFeaturesLeftQueue.get().trackedFeatures: outputFeaturesLeftQueueGet a feature tracking result object from the output feature queue named and extract the tracked features from it.

2. drawFeatures(leftFrame, trackedFeaturesLeft): Call drawFeaturesthe function named and pass in the left frame leftFrameand tracked left features trackedFeaturesLeftas parameters to draw feature points on the left frame.

3. trackedFeaturesRight = outputFeaturesRightQueue.get().trackedFeatures: outputFeaturesRightQueueGet another feature tracking result object from the output feature queue named , and extract the tracked features from it.

4. drawFeatures(rightFrame, trackedFeaturesRight): Call drawFeaturesthe function named and pass in the right frame rightFrameand tracked right features trackedFeaturesRightas parameters to draw feature points on the right frame.

display frame image

        cv2.imshow(leftWindowName, leftFrame)
        cv2.imshow(rightWindowName, rightFrame)

Use cv2.imshowthe function of OpenCV to display the left and right frames.

1. cv2.imshow(leftWindowName, leftFrame): Use OpenCV's cv2.imshowfunction to display the left frame. leftWindowNameis the name of the window on the left, used to identify the window, leftFrameand is the frame to display.

2. cv2.imshow(rightWindowName, rightFrame): Use OpenCV's cv2.imshowfunction to display the right frame. rightWindowNameis the name of the window on the right, used to identify the window, rightFrameand is the frame to display.

Responding to keyboard input

        key = cv2.waitKey(1)
        if key == ord('q'):
            break
        elif key == ord('s'):
            if featureTrackerConfig.cornerDetector.type == dai.FeatureTrackerConfig.CornerDetector.Type.HARRIS:
                featureTrackerConfig.cornerDetector.type = dai.FeatureTrackerConfig.CornerDetector.Type.SHI_THOMASI
                print("Switching to Shi-Thomasi")
            else:
                featureTrackerConfig.cornerDetector.type = dai.FeatureTrackerConfig.CornerDetector.Type.HARRIS
                print("Switching to Harris")

            cfg = dai.FeatureTrackerConfig()
            cfg.set(featureTrackerConfig)
            inputFeatureTrackerConfigQueue.send(cfg)

The above code implements waiting for the user to press a key on the keyboard, and performs different operations according to the key pressed.

1. key = cv2.waitKey(1): Wait for the user to press a key on the keyboard, 1which means waiting for 1 millisecond. This function returns the ASCII value of the key.

2. if key == ord('q'):: If the key pressed is qthe ASCII value corresponding to the letter, that is, the user presses qthe key:

   • break: Jump out of the loop and end the program.

3. elif key == ord('s'):: If the key pressed is sthe ASCII value corresponding to the letter, that is, the user presses sthe key:

   • if featureTrackerConfig.cornerDetector.type == dai.FeatureTrackerConfig.CornerDetector.Type.HARRIS:: Check whether the type of the current corner detector is Harris corner detector.

     • featureTrackerConfig.cornerDetector.type = dai.FeatureTrackerConfig.CornerDetector.Type.SHI_THOMASI: If it is a Harris corner detector, switch the corner detector type to Shi-Thomasi corner detector.

       • print("Switching to Shi-Thomasi"): Print a message indicating that a switch to the Shi-Thomasi corner detector has been made.

     • else:: If the type of the current corner detector is not Harris corner detector.

       • featureTrackerConfig.cornerDetector.type = dai.FeatureTrackerConfig.CornerDetector.Type.HARRIS: Switches the corner detector type back to the Harris corner detector.

       • print("Switching to Harris"): Print a message indicating that the Harris corner detector has been switched back.

   • cfg = dai.FeatureTrackerConfig(): Create a new dai.FeatureTrackerConfigobject.

   • cfg.set(featureTrackerConfig): Copies the current feature tracker configuration featureTrackerConfiginto the newly created object.

   • inputFeatureTrackerConfigQueue.send(cfg): Sends the feature tracker configuration object to the named inputFeatureTrackerConfigQueuequeue.

Setup 12: Run the program

Enter the following command in the terminal to run the program

python main.py

Run the program and see the effect