1 ray casting
Both ray casting (ray diffraction) and ray tracing (ray tracing) are basic methods in computer graphics; used for three-dimensional solid modeling and image rendering; the word ray casting comes from General Motors Research Labs from 1978–1980; from Paper "Ray Casting for Modeling Solids". ray casting greatly simplifies the rendering of 3D objects or scenes to images. See the wiki for details .
Here is a brief introduction to the following ray casting; under the assumption that the light will not be reflected twice or multiple times, from the focal point of the camera as the center point, each pixel is calculated according to the set FOV to obtain the corresponding light direction on each pixel, and Shoot all rays from this direction. If the rays intersect with objects in the scene, record them, and select the object with the shortest distance from the camera as the result of the rays and map them to the image. In short: Given a scene and a camera pose, ray casting can be used to obtain image information captured in that pose .
The following are several schematic diagrams of online ray casting
The RaycastingScene class is implemented in open3d for the basic ray casting function; the following describes how to create a simple scene in open3d and use ray casing to perform the intersection test of light and objects; at the same time, this class can be used as a mesh; cad model Create virtual point cloud information.
import open3d as o3d
import numpy as np
import matplotlib.pyplot as plt
if __name__ == "__main__":
# Create meshes and convert to open3d.t.geometry.TriangleMesh .
# 创建普通的geometry类型在cpu上
# 创建立方体
cube_ = o3d.geometry.TriangleMesh.create_box().translate([0, 0, 0])
# 创建环面
torus_ = o3d.geometry.TriangleMesh.create_torus().translate([0, 0, 2])
# 创建球体
sphere_ = o3d.geometry.TriangleMesh.create_sphere(radius=0.5).translate(
[1, 2, 3])
o3d.visualization.draw_geometries([cube_, torus_, sphere_],
lookat=[0, 0, 2], # 相机查看的方向
front=[2, 3, 0], # 相机的位置
up=[0, 1, 0],
zoom=1.0,
)
# 将cpu上的geometry类型转换成tensorgeometry的类型，可以运算在显卡等设备上加速处理
# 由于ray casting操作运算量大，所以open3d没有基于cpu对该类进行实现，
# 但是open3d做了对集成显卡的支持
cube = o3d.t.geometry.TriangleMesh.from_legacy(cube_)
sphere = o3d.t.geometry.TriangleMesh.from_legacy(sphere_)
torus = o3d.t.geometry.TriangleMesh.from_legacy(torus_)
# 初始化RaycastingScene类
# param : nthreads用于指定使用多少个线程用于创建场景，0代表自动选择
scene = o3d.t.geometry.RaycastingScene(nthreads=0)
# 添加上述创建的物体到场景中，返回值为该geometry在RaycastingScene中的ID
_ = scene.add_triangles(cube)
_ = scene.add_triangles(torus)
_ = scene.add_triangles(sphere)
"""
生成的光线示例
# We create two rays:
# The first ray starts at (0.5,0.5,10) and has direction (0,0,-1).
# The second ray start at (-1,-1,-1) and has direction (0,0,-1).
rays = o3d.core.Tensor([[0.5, 0.5, 10, 0, 0, -1], [-1, -1, -1, 0, 0, -1]],
dtype=o3d.core.Dtype.Float32)
"""
# 根据相机的参数设置光线的射出方向
# 返回值是一个大小为（height_px，width_px，6）的张量为
# [ox, oy, oz, dx, dy, dz] 中心坐标时[ox,oy,oz] 方向为 [dx,dy,dz]
rays = o3d.t.geometry.RaycastingScene.create_rays_pinhole(
fov_deg=90, # 相机的水平fov角度
center=[0, 0, 2], # 相机的位置
eye=[2, 3, 0], # 相机查看的方向
up=[0, 1, 0],
width_px=640, # 图像的宽度
height_px=480, # 图像的高度
)
# We can directly pass the rays tensor to the cast_rays function.
# 计算光线与场景中的物体第一次相交
# 输入的rays的维度大于等于2，shape为{..,6}，
# 方向向量不必进行归一化操作，但是返回的最近的物体与相机的距离结果为方向单位向量的长度
ans = scene.cast_rays(rays=rays, nthreads=0)
"""
print(ans) 结果为
dict_keys(['primitive_uvs', 'primitive_ids', 'geometry_ids', 'primitive_normals', 't_hit'])
t_hit is the distance to the intersection. The unit is defined by the length of the ray direction. If there is no intersection this is inf
t_hit 是光线与物体相交点距离相机的距离，单位是光线方向的单位向量，如果该光线没有与物体相交，则该值为inf
geometry_ids gives the id of the geometry hit by the ray. If no geometry was hit this is RaycastingScene.INVALID_ID
geometry_ids 给出了该光线与哪个ID的物体相交，如果该光线没有与物体相交，则该值为RaycastingScene.INVALID_ID
primitive_ids is the triangle index of the triangle that was hit or RaycastingScene.INVALID_ID
primitive_ids 返回了光线与triangle mesh的哪个mesh相交，返回了该mesh在triangle mesh中的索引，
如果该光线没有与物体相交，则该值为RaycastingScene.INVALID_ID
primitive_uvs is the barycentric coordinates of the intersection point within the triangle.
primitive_uvs 是三角形内交点的重心坐标 shape：{.., 2}
primitive_normals is the normal of the hit triangle.
primitive_normals 相交triangle mesh的法线信息 shape： {.., 3}
"""
# 使用matplotlib画出所有光线距离相机的图像，颜色越深代表距离越远
plt.imshow(ans['t_hit'].numpy())
plt.show()
# 使用matplotlib画出所有光线对应mesh的法线信息
plt.imshow(np.abs(ans['primitive_normals'].numpy()))
plt.show()
# 使用matplotlib画出所有光线对应的物体id，不同的物体颜色不同，相同的物体颜色相同
plt.imshow(np.abs(ans['geometry_ids'].numpy()), vmax=3)
plt.show()
Note: For the parameters of the camera angle of view, you can refer to the relevant content of opengl. Here is a tutorial link LearnOpenGL - Camera
Depth map of camera and object
The normal information of the surface of the object
Each pixel represents a ray, and when the ray intersects with an object, it is uniformly displayed as the same color
2 Create a virtual point cloud using ray casting
Combining the above content, the point cloud information of the object can be created according to the depth information obtained by the intersection of the light and the object. In the virtual environment, the virtual camera pose information can be used to obtain the object seen by the camera in the current pose through raycasting. depth map.
# 判断该点是否与物体相交
hit = ans['t_hit'].isfinite()
# 取出所有与物体相交的rays，并根据结果t_hit来获取每个相交光线与物体的深度距离，加在原光线点中
points = rays[hit][:,:3]
points_distance = rays[hit][:,3:]*ans['t_hit'][hit].reshape((-1,1))
points+=points_distance
pcd = o3d.t.geometry.PointCloud(points)
# Press Ctrl/Cmd-C in the visualization window to copy the current viewpoint
o3d.visualization.draw_geometries([pcd.to_legacy()],
front=[0.5, 0.86, 0.125],
lookat=[0.23, 0.5, 2],
up=[-0.63, 0.45, -0.63],
zoom=0.7)
# o3d.visualization.draw([pcd]) # new API
3 Using ray casting in TSDF
This content is supplemented when rebuilding TSDF