Depth Camera First Experience: Hello World

When my team leader gave me a depth camera, the unlucky things started. Some problems encountered in the process of using it could not be found, and my head was bald.

Configuration:
ubuntu20.04 (I'm really too lazy to upgrade, once I upgrade, there may be a possibility that I can't find the tutorial)
In addition, this system has been broken by me, and I don't remember how many strange things I downloaded into it , so I was numb when reporting an error
Depth camera: OAK-D-PRO

Here is just to test how to use the camera according to the official tutorial, specifically here

1. Confirm dependencies

Python 3.6 (Ubuntu) or Python 3.7 (Raspbian)。

安装或升级 DepthAI Python API安装详解

需要用到 cv2 和 numpy Python 模块。

1.1 Test the python environment:

python3 --version

insert image description here

1.2 Install or upgrade DepthAI Python API

Here is a reference to this blog
. Make sure your depth camera is connected before running the command.

lsusb | grep MyriadX

insert image description here

Run the following command in the terminal to clone the deephai repository

git clone https://github.com/luxonis/depthai.git

insert image description here

Run the cd depthai command in the terminal to change directory to this directory

Run the following command on the terminal to install dependencies

python3 install_requirements.py

Run the following command in Terminal to run the demo script from within DepthAI to make sure everything works

python3 depthai_demo.py

insert image description here
This proves that you have succeeded.

1.3 The last dependency is generally not missing

Two, start the configuration file

2.1 Set up the following file structure on your computer:

cd ~
mkdir -p depthai-tutorials-practice/1-hello-world
touch depthai-tutorials-practice/1-hello-world/hello-world.py
cd depthai-tutorials-practice/1-hello-world

2.2 Install pip dependencies

To display the DepthAI color video stream, we need to import a small number of packages. Download and install the packages required for this tutorial:

python3 -m pip install numpy opencv-python depthai --user -i https://pypi.tuna.tsinghua.edu.cn/simple

insert image description here

2.3 Test your environment

Let's verify that all dependencies are loaded. Open the hello-world.py file you created earlier in your code editor. Copy and paste the following into hello-world.py:

import numpy as np # numpy - manipulate the packet data returned by depthai
import cv2 # opencv - display the video stream
import depthai # access the camera and its data packets

Try running the script and make sure it executes without errors:

python3 hello-world.py

Confirm that it is correct and start the next step

2.4 start hello world

The specific steps can be seen here , only the specific code in hello-world.py is pasted here

# first, import all necessary modules
from pathlib import Path

import blobconverter
import cv2
import depthai
import numpy as np

# Pipeline tells DepthAI what operations to perform when running - you define all of the resources used and flows here
pipeline = depthai.Pipeline()

# First, we want the Color camera as the output
cam_rgb = pipeline.createColorCamera()
cam_rgb.setPreviewSize(300, 300)  # 300x300 will be the preview frame size, available as 'preview' output of the node
cam_rgb.setInterleaved(False)

# Next, we want a neural network that will produce the detections
detection_nn = pipeline.createMobileNetDetectionNetwork()
# Blob is the Neural Network file, compiled for MyriadX. It contains both the definition and weights of the model
# We're using a blobconverter tool to retreive the MobileNetSSD blob automatically from OpenVINO Model Zoo
detection_nn.setBlobPath(blobconverter.from_zoo(name='mobilenet-ssd', shaves=6))
# Next, we filter out the detections that are below a confidence threshold. Confidence can be anywhere between <0..1>
detection_nn.setConfidenceThreshold(0.5)
# Next, we link the camera 'preview' output to the neural network detection input, so that it can produce detections
cam_rgb.preview.link(detection_nn.input)

# XLinkOut is a "way out" from the device. Any data you want to transfer to host need to be send via XLink
xout_rgb = pipeline.createXLinkOut()
# For the rgb camera output, we want the XLink stream to be named "rgb"
xout_rgb.setStreamName("rgb")
# Linking camera preview to XLink input, so that the frames will be sent to host
cam_rgb.preview.link(xout_rgb.input)

# The same XLinkOut mechanism will be used to receive nn results
xout_nn = pipeline.createXLinkOut()
xout_nn.setStreamName("nn")
detection_nn.out.link(xout_nn.input)

# Pipeline is now finished, and we need to find an available device to run our pipeline
# we are using context manager here that will dispose the device after we stop using it
with depthai.Device(pipeline) as device:
    # From this point, the Device will be in "running" mode and will start sending data via XLink

    # To consume the device results, we get two output queues from the device, with stream names we assigned earlier
    q_rgb = device.getOutputQueue("rgb")
    q_nn = device.getOutputQueue("nn")

    # Here, some of the default values are defined. Frame will be an image from "rgb" stream, detections will contain nn results
    frame = None
    detections = []

    # Since the detections returned by nn have values from <0..1> range, they need to be multiplied by frame width/height to
    # receive the actual position of the bounding box on the image
    def frameNorm(frame, bbox):
        normVals = np.full(len(bbox), frame.shape[0])
        normVals[::2] = frame.shape[1]
        return (np.clip(np.array(bbox), 0, 1) * normVals).astype(int)


    # Main host-side application loop
    while True:
        # we try to fetch the data from nn/rgb queues. tryGet will return either the data packet or None if there isn't any
        in_rgb = q_rgb.tryGet()
        in_nn = q_nn.tryGet()

        if in_rgb is not None:
            # If the packet from RGB camera is present, we're retrieving the frame in OpenCV format using getCvFrame
            frame = in_rgb.getCvFrame()

        if in_nn is not None:
            # when data from nn is received, we take the detections array that contains mobilenet-ssd results
            detections = in_nn.detections

        if frame is not None:
            for detection in detections:
                # for each bounding box, we first normalize it to match the frame size
                bbox = frameNorm(frame, (detection.xmin, detection.ymin, detection.xmax, detection.ymax))
                # and then draw a rectangle on the frame to show the actual result
                cv2.rectangle(frame, (bbox[0], bbox[1]), (bbox[2], bbox[3]), (255, 0, 0), 2)
            # After all the drawing is finished, we show the frame on the screen
            cv2.imshow("preview", frame)

        # at any time, you can press "q" and exit the main loop, therefore exiting the program itself
        if cv2.waitKey(1) == ord('q'):
            break

The effect is as follows:
insert image description here

3. Summary

Due to luck, I have been reporting errors before, and suddenly it can run. I am so happy