Intel Realsense d435 uses python to preprocess depth maps

Intel Realsense d435 uses python to preprocess depth maps

This article mainly translates an article about depth map preprocessing filters, and will also talk about distance measurement later.
The image display in the original article uses the matplotlib.pyplot tool. In this article, opencv is used to display the depth map.
First, import the package by regular operations:

import cv2
import numpy as np
import pyrealsense2 as rs

Get the images we need, either through the camera or locally

pipe = rs.pipeline()
cfg = rs.config()
cfg.enable_stream(rs.stream.depth,640,480,rs.format.z16,30)
cfg.enable_stream(rs.stream.color,640,480,rs.format.bgr8,30)
profile = pipe.start(cfg)

try:
    while True:
        frame = pipe.wait_for_frames()
        depth_frame = frame.get_depth_frame()
        print('capture success')
        if cv2.waitKey(10)&0xff == ord('q'):
            break

finally:
    pipe.stop()

Obtain successfully, print information in the terminal

1. Visualization of depth images

Use the colorizer provided in the pyrealsense2 library to convert the depth value in the image into a displayable form:

colorizer = rs.colorizer()
colorizer_depth = np.asanyarray(colorizer.colorize(depth_frame).get_data())
cv2.imshow('colorizer depth',colorizer_depth)

2. Apply filters

(1) Decimation

When using Depth-from-Stereo solution, z-accuracy is related to original spatial resolution.
If you are satisfied with lower spatial resolution, the Decimation Filter will reduce spatial resolution preserving z-accuracy and performing some rudamentary hole-filling.
This passage It means that it is similar to downsampling in image processing. The resolution of the image will be reduced (640x480----->320x240), but some black holes will be filled.

 		#二、抽取Decimation
        colorizer = rs.colorizer()
        decimation = rs.decimation_filter()
        decimationed_depth = decimation.process(depth_frame)
        colorized_depth = np.asanyarray(colorizer.colorize(decimationed_depth).get_data())
        cv2.imshow('decimationed depth',colorized_depth)

        #可以设置参数类似于迭代次数
        decimation.set_option(rs.option.filter_magnitude,4)
        decimationed_depth = decimation.process(depth_frame)
        colorized_depth = np.asanyarray(colorizer.colorize(decimationed_depth).get_data())
        cv2.imshow('decimationed4 depth',colorized_depth)

(2) Spatial Filter

Paper on spatial filters:
Domain Transform for Edge-Aware Image and Video Processing

#三、空间滤波器（spatial filter）
        colorizer = rs.colorizer()
        spatial = rs.spatial_filter()
        filtered_depth = spatial.process(depth_frame)
        colorized_depth = np.asanyarray(colorizer.colorize(filtered_depth).get_data())
        cv2.imshow('filtered depth',colorized_depth)

        # 可以设置相关的参数
        spatial.set_option(rs.option.filter_magnitude,5)
        spatial.set_option(rs.option.filter_smooth_alpha,1)
        spatial.set_option(rs.option.filter_smooth_delta,50)
        filtered_depth = spatial.process(depth_frame)
        colorized_depth = np.asanyarray(colorizer.colorize(filtered_depth).get_data())
        cv2.imshow('filtered1 depth',colorized_depth)

At the same time, spatial also provides a basic hole filling operation:

 		spatial.set_option(rs.option.holes_fill,3)
        filtered_depth = spatial.process(depth_frame)
        colorized_depth = np.asanyarray(colorizer.colorize(filtered_depth).get_data())
        cv2.imshow('fill hole',colorized_depth)

(3) Temporal Filter

Our implementation of Temporal Filter does basic temporal smoothing and hole-filling . It doesn't make sense when applied to a single frame, so let's capture several consecutive frames:

 #四、时间滤波器（Temporal Filter）
        colorizer = rs.colorizer()
        
        frames.append(depth_frame)
        i += 1
        if i == 10:
            i = 0
            temporal = rs.temporal_filter()
            print(len(frames))
            for x in range(10):
                temp_filtered = temporal.process(frames[x])
            frames = []
            colorized_depth = np.asanyarray(colorizer.colorize(temp_filtered).get_data())
            cv2.imshow('temporal depth',colorized_depth)

(4)Hole Filling

Hole Filling filter offers additional layer of depth exterpolation:
Hole Filling filter offers additional layer of depth extrapolation:

#五、孔填充（Hole Filling）
        colorizer = rs.colorizer()
        hole_filling = rs.hole_filling_filter()
        filled_depth = hole_filling.process(depth_frame)
        colorized_depth = np.asanyarray(colorizer.colorize(filled_depth).get_data())
        cv2.imshow('filled depth',colorized_depth)

(5) Putting Everything Together

These filters work best when applied sequentially one after another. At longer range, it also helps using disparity_transform to switch from depth representation to disparity form:
These filters work best when applied sequentially one after another. On a larger scale, it also helps to use disparity_transform to convert from a depth representation to a disparity form:

#六、复合多个滤波器
        colorizer = rs.colorizer()        
        depth_to_disparity = rs.disparity_transform(True)
        disparity_to_depth = rs.disparity_transform(False)

        decimation = rs.decimation_filter()
        spatial = rs.spatial_filter()
        temporal = rs.temporal_filter()
        hole_filling = rs.hole_filling_filter()


        frames.append(depth_frame)
        i += 1
        if i == 10:
            i = 0
            for x in range(10):
                frame = frames[x]
                frame = decimation.process(frame)
                frame = depth_to_disparity.process(frame)
                frame = spatial.process(frame)
                frame = temporal.process(frame)
                frame = disparity_to_depth.process(frame)
                frame = hole_filling.process(frame)
            frames = []
            colorized_depth = np.asanyarray(colorizer.colorize(frame).get_data())
            cv2.imshow('more depth filter',colorized_depth)