Author：Charles R. Qi Li Yi Hao Su Leonidas J. Guibas Stanford University
会议：Conference on Neural Information Processing Systems (NIPS) 2017
一、回顾PointNet
原理：
1、learn a spatial encoding of each point
2、aggregate all individual point features to a global point cloud signature（a single max pooling）
优点：an effective architecture to process an unordered set of points for semantic feature extraction
缺点：
1、does not capture local structure
2、没有考虑到不同的密度
方法：
point set {x1, x2, ..., xn}
γ and h are usually multi-layer perceptron (MLP) networks
h can be interpreted as the spatial encoding of a point
二、PointNet++
原理：
1、We first partition the set of points into overlapping local regions by the distance metric of the underlying space.
2、Similar to CNNs, we extract local features capturing fine geometric structures from small
neighborhoods; such local features are further grouped into larger units and processed to produce higher level features.
3、This process is repeated until we obtain the features of the whole point set
问题：
1、如何划分点 -> 确定每个划分的中心和距离
centroids（图心） are se lected among input point set by a farthest point sampling (FPS) algorithm
2、如何提取局部特征——PointNet.
三、问题设置
X = (M, d)：M ⊆ R^n is the set of points and d is the distance metric
学习一个输入为X的映射f，结果可以是分类、分割
四、方法
composed by a number of set abstraction levels
At each level, a set of points is processed and abstracted to produce a new set with fewer elements
1、set abstraction level
组成部分：
1）Sampling layer
作用：selects a set of points from input points, which defines the centroids of local regions
. 方法：iterative farthest point sampling (FPS)
具体操作：Given input points {x1, x2, ..., xn}，choose a subset of points {xi1, xi2, ..., xim}, such that xij is the most distant point (in metric distance) from the set {xi1, xi2, ..., xij−1} with regard to the rest points
2）Grouping layer
作用：constructs local region sets by finding “neighboring” points around the centroids.
具体操作：输入N × (d + C)的点集合和N‘ × d的图中心坐标集合，输出N’ × K × (d + C)，K is the number of points in the neighborhood of centroid points（每个region K都不一样）。使用ball query’s local neighborhood找K个点
3）PointNet layer
作用：uses a mini-PointNet to encode local region patterns into feature vectors
具体操作：对于每一local region，输入N’ × K × (d + C)，输出N’ × (d + C‘)。在计算中将此region的坐标转换为x=x-xc，减去中心点的坐标，再输入到Point net中
输入输出：
输入：N × (d + C) matrix：N points with d-dim coordinates and C-dim point feature
输出：N' × (d + C’) matrix of N‘ subsampled points with d-dim coordinates and new C‘-dim feature vectors summarizing local context
2、Robust Feature Learning under Non-Uniform Sampling Density
Multi-scale grouping (MSG)
思想：apply grouping layers with different scales followed by according PointNets to extract features of each scale. Features at different scales are concatenated to form a multi-scale feature.
方法：randomly dropping out input points：1）for each training point set, we choose a dropout ratio θ uniformly
sampled from [0, p] where p ≤ 1。2）drop a point with probability θ
During test, we keep all available points
Multi-resolution grouping (MRG)
两个组成部分：1）summarizing the features at each subregion from the lower level Li−1 using
the set abstraction level；2）directly processing all raw points in the local region using a single PointNet.
两个部分的权重设置：1）the density of a local region is low，the second vector should be weighted higher；2）the density of a local region is high, the first vector weighted higher
3、Segmentation
1）interpolation：使用inverse distance weighted average based on KNN
2）将插值产生的feature concatenated with skip linked point features
3）通过“unit pointnet”（类似CNN中的1*1卷积）进行特征降维
五、实验
1、Datasets
• MNIST:
2D。60k training and 10k testing
classification：选取512 points
• ModelNet40:
3D objects。CAD models of 40 categories。9,843 shapes for training and 2,468 for testing.
classification：选取1024 points
• SHREC15: 3D objects。1200 shapes from 50 categories。 We use five fold cross validation。
• ScanNet: 3D scenes。1513 scanned and reconstructed indoor scenes。1201 scenes for training, 312 scenes for test.
2、Point Set Classification in Euclidean Metric Space
效果：1）比point net好；2）有时比image CNNs好
鲁棒性测试：randomly drop points during test time to validate our network’s robustness to non-uniform and sparse data
3、Point Set Classification in Non-Euclidean Metric Space
针对问题：同一个类别，但是pose不一样，如一匹马可能直立可能弯着。这两者应有一样的分类结果
方法步骤：1）construct the metric space induced by pairwise geodesic distances。2）obtain an embedding metric。3）extract intrinsic point features in this metric space including WKS [1], HKS [27] and multi-scale Gaussian curvature。4）use these features as input and then sample and group points according to the underlying metric space
4、Segmentation
