How to implement the hidden point removal algorithm of Open3D? Point Cloud Hidden Point Removal Algorithm——A Super-Detailed Implementation of the Underlying Algorithm [In-depth understanding of the essence of the algorithm]

written in front

The hidden point removal algorithm (Hidden Point Removal) in Open3D is used to remove points in the background that are not occluded by other points when rendering a point cloud from a given viewpoint, thereby improving the visualization of the point cloud. A hidden point removal method based on the approximate calculation of point cloud visibility proposed by [Katz2007] is implemented in the Open3D library. This method does not require surface reconstruction or normal vector estimation, and directly uses point cloud data to calculate the visibility of points.
When using the Open3D library we only need one line
_, pt_map = pcd.hidden_point_removal(camera, radius)
to get from the original point cloud

Get the point cloud after removing hidden points

But how does this algorithm achieve occluded point removal in point clouds where point occlusions are almost non-existent?
I found this document Direct Visibility of Point Sets to understand the underlying logic of the code. Later, many people optimized this article on the basis of it, but it remains the same. The algorithm of the article will be explained below.

Interpretation of the article

introduce

The paper introduces a new approach - extracting visibility information directly from point clouds. Point cloud is a set of sampling points containing 3D position information, which is simpler and more flexible than traditional grid representation. By extracting the visibility information in the point cloud, it is possible to realize the visualization of point sets, view-dependent fast reconstruction and real-time shadow casting. The method proposed in this paper is simple and fast, does not depend on screen resolution and mesh topology, and is applicable to point clouds of various dimensions. In addition, the algorithm is proved theoretically, and the experimental results and application examples are given.

Algorithm implementation

Given a point set P = { pi ∣ 1 ≤ i ≤ n } ⊂ ℜ DP = \{p_{i}|1\le{i}\le{n}\}\sub{\Re^{D}}P={ pi​∣1≤i≤n}⊂ℜD , these points are on a continuous surface$S$ upsampled. and a viewpoint$C.$ _ Our goal is to findallAll point sets PPthat can be seen on$C$points in$P.$

This kind of problem is that almost all points are visible due to the sparsity of the point cloud, so the simple line visibility judgment cannot be applied to all situations. So there must be a new point cloud visibility definition method. If a point is within radius $\rho$ 's open ball

In mathematics, a point pprrof$p$$r$ radius open sphere$B_r\left(p\right)$ means in$p$ is the center of the sphere,rrA sphere of radius$r\; ,\; but\; not\; containing\; any\; points\; on\; the\; surface\; of\; the\; sphere.$That is,$B_r\left(p\right)$ contains all distances$p$ is less than$r$ , but not points on the surface of the sphere.

A small disturbance at any position within will not be occluded by other points, then this point is visible.
Clearly, a reasonable criterion for visibility must be related to sampling density. Suppose $P$ is planeSSAρ − \rho-$of\; S$$\rho -$ sampling, that is, if we use a radius of$\rho$ 's open sphere to enclose each sampling point$p_i$Then surface $S$ will be completely contained within the union of these spheres. If we perturb pi p_ianywhere on the ball$p_i$The positions of which are not blocked by other points, then $p_i$Just visible. But this algorithm works well only when the line of sight is perpendicular to the surface, and does not perform well when the plane is inclined. But we also have to avoid computing surface normals.
To solve this problem, the authors propose an operator named Hidden Point Removal (HPR), which has the characteristics of correctness, suitability for inclined surfaces and reasonable time complexity. This operator consists of two steps: Inversion and Convex Hull Construction.

inversion (point transformation)

Given the point set $P$ and$C$ , we will$P$ is associated with a coordinate system, let$C$ is placed at the origin. Then we need to find a function that will point$p_i$map to CC along$C$ to$p_i$on the ray, and at $\Vert p_i\Vert$ is monotonically decreasing. where$\Vert \cdot \Vert$ is a norm. This paper introduces an inversion method called Spherical Flipping, which converts a D-dimensional sphere with radius$R$ center is at$The\; spheres\; of\; C$ are connected, and this sphere containsPPAll points of$P.$Use the following formula to$p_i$Make a mirror reflection.
$${\hat{p}}_i=f(p_i)=p_i+2(R-\Vert p_i\Vert )\frac{p_i}{\Vert p_i\Vert }$$
The schematic diagram of mirror reflection is as follows. This formula realizes the mirror reflection of the blue cat from the green sphere to the red cat in the outer circle.

The author hinted that there are more than one inversion function, such as $\tilde{f}(p_i)=\frac{p_i}{\Vert p_i\Vert \gamma }$This formula can also achieve a similar effect.

Convex hull structure

The convex hull of a point cloud is the smallest convex set that contains all points. Open3D includes methods for computing the convex hull of point clouds. The implementation is based on Qhull. Using the function compute_convex_hull, you can get the visual convex hull of the following figure:

But how is this convex hull constructed?
Through the point transformation in the previous step, we can get the transformed point cloud $\hat{P}$ , the convex hull we want to calculate is the transformed point cloud$\hat{P}$ and center pointCCThe union of$C.$
The main result in the article is to extract the location at$\hat{P}\cup \left\{C\right\}$ convex hull to be equivalent to visible points. CChere$The\; introduction\; of\; C$ is necessary, because if C is outside the point cloud, the points on the back of the object will be on the convex hull.
We can thus derive a definition:

If point $p_i$The reflection point $\widehat{p_i}$at $\hat{P}\cup On\; the\; convex\; hull\; of\; \{C\}$ , then point$p_i$is marked as from $C$ is visible.

The HPR operator can be applied to any dimension and is easier to understand in two-dimensional space. When we consider a two-dimensional point set $p_i\in P.$ _ Without loss of generality,$p_i$placed on the X-axis. Use the polar coordinate system $(r,\theta )$ , we can put$P_i$Mapping $p_i=(r_i,0)$ . ri r_iin that$r_i$is $p_i$to $The\; distance\; between\; C$ and the angle with the X axis is 0. Consider passing through$\widehat{p_i}$and forms an angle β \beta with the X axisThe straight line L ^ \hat{L}of$\beta$$\hat{L}$ , as shown in the figure below:

we want to find$\hat{L}$ The line L corresponding to spherical flipping$L=\left(r\right(a),\alpha )$ , using the law of sines we can get:
$$\frac{2R-r_i}{\sin (p-a-b)}=\frac{R-r(a}{\sin b}$$
thus we get
$$L=\left(r\right(a),a)=(2R+\frac{(r_i-2R)\_\sin }{\sin (a+b)},\alpha )$$
Curve$L$ passes through$p_i$and $C$ , needs to be represented by a quartic polynomial of x and y in the Cartesian coordinate system.
The figure below illustrates$How\; does\; the\; shape\; of\; L$ vary with the angle$\beta$ changes.

$The\; L$ and X axes define the points$p_i$associated white space. $p_i$Visibility depends on the size of the area. The larger the area is, $p_i$more visible. The important property of HPR is that this size is given by $p_i$The surrounding points are adaptively determined as follows:
For all given points $p_i$, P P There are always two special pointspj p_j$on\; both\; sides\; of\; P$$p_j$和$p_k\in P$ , their corresponding curve$L_j=(r_j(a_j),a_j)$和$L_k=\left(r_k\right(a_k),a_k)$ , the area between them is the largest possible blank area, as shown in the figure below:

当$p_i$when visible

when $p_i$When invisible

From the formula of the curve, it can be deduced that the largest area corresponds to the smallest $beta$ . This means that for$p_i$
In terms of β j \beta_j corresponding to the largest possible blank area$b_j$and $b_k$the smallest.
$b_j$and $b_k$Available from LLExtracted from the equation of$L.$For visible$p_i$，$b_j$and $b_k$The sum of should satisfy $b_j+b_k\le const$ .$\_$
set$const=\pi$ means that it is computationally unnecessary to find the adjacent points$p_j、p_k$up. Instead, computing $\hat{P}\cup The\; convex\; hull\; of\; \{C\}$ is sufficient. This is because if and only if$\hat{P}\cup \left\{C\right\}$ All points located by$\overline{\hat{p_i},\hat{p_j}}$and $\overline{\hat{p_i},\hat{p_k}}$When defining one side of the half-space, the point $\widehat{p_i}$Only at $\hat{P}\cup On\; the\; convex\; hull\; of\; \{C\}$ . In the case of the text, since$L_j$and $L_k$The area between is empty (the gray area in the figure), so $\widehat{L_j}$and $\widehat{L_k}$The far field must be empty.
This observation is computationally important and is the reason why the HPR operator is so efficient, if you don't compute the convex hull you have to find $p_j$and $p_k$Make the algorithm quadratic (or even cubic in 3D). Instead, if all that needs to be done is to compute the convex hull and consider if $\widehat{p_i}$at $P$Pointpi p_i$\stackrel{on\; the\; convex\; hull\; of\; ^}{}$$p_i$is visible on it. The HPR operator thus defines the shape and size of the empty region.
The above description can be extended to 3D, in which case the empty regions are not defined by curves but by surfaces. Also, instead of two neighbors in 3D, at least three neighbors define the surface that encloses the empty volume, but the idea of ​​this convex hull solution is the same.
It is worth noting that even if we define $\pi$ is a constant threshold, and the visibility threshold can also be modified indirectly by changing the radius of the sphere,RRLarger$R$$The\; shape\; of\; the\; L$ will change.