Article directory
Preface
paper address https://openreview.net/pdf?id=Bkl-43C9FQ
AuthorsChiyu “Max” Jiang , Jingwei Huang, Karthik Kashinath, Prabhat, Philip Marcus, Matthias Niessner
github address https://github.com/maxjiang93/ugscnn
1. Introduction
Traditional convolutional neural network weather forecasting is done on maps generated under Mercator Projection, which results in excess information in the polar regions, while sparse sampling in the equatorial regions results in insufficient information.
To solve this problem, Chiyu “Max” Jiang et al. proposed using spherical convolution to improve traditional convolution. This has to introduce a new problem - how to construct three-dimensional grid data based on spherical convolution. The article argues that a large number of machine learning problems in computer vision and related fields require processing signals in the spherical domain; for example, omnidirectional RGBD images from commercial panoramic cameras, panoramic videos coupled with lidar scans of autonomous vehicles, or in scientific fields planetary signals). Unfortunately, simple mapping of spherical signals to planar domains results in image distortion. In particular, projection artifacts and boundary processing near the polar regions make the learning of 2D convolutional neural networks (CNN) particularly challenging. The 2018 work proposed a network architecture that operates natively in the spherical domain and is rotation invariant in the SO(3) group. This invariance is desirable in a range of problems - for example, machine learning problems with molecules - where gravitational effects are negligible and the direction is arbitrary. However, for other different classes of problems, the hypothesized direction information is crucial to the network's predictive ability. The MNIST digit recognition problem is a good example of this, where orientation plays an important role in distinguishing the digits "6" and "9". Other examples include omnidirectional images, where the image is naturally oriented by gravity; and planetary signals, where planets are naturally oriented around their axis of rotation.
The article proposes a new convolution kernel for CNN on arbitrary manifolds and topologies, discretized by unstructured grids (i.e. grids), and focuses on its performance in the spherical domain approximated by the icosahedral spherical grid. application. The article proposes and evaluates the use of a new parameterization scheme for CNN convolution kernels, called parametric differential operators (PDOs), which can be easily implemented on unstructured grids. We call convolution operators that operate on meshes using such kernels MeshConv operators. This parameterization scheme uses only 4 parameters per core and achieves better performance than other methods with much fewer parameters.
2. Related technologies
1.Spherical CNNs
When dealing with spherical signals, the first thing to consider is the distortion that occurs when the signal is projected from a curved surface to a flat surface. Su and Grauman processed equirectangular images with regular convolutions that increased the kernel size near the polar regions, and planar mapping introduced greater distortion in the polar regions. Coors et al. and Zhao et al. use a constant kernel to sample points on the tangent plane of the spherical image to reduce distortion. A slightly different literature explores rotationally equivariant implementations of spherical CNNs. Cohen et al. proposed spherical convolution with rotationally equivariant intermediate feature maps in SO(3). Esteves et al. used a spherical harmonic base to achieve similar results.
2.Reparameterized Convolutional Kernel
Re-parameterization of the convolution kernel is beneficial to the characterization of convolution kernels that are difficult to solve or difficult to discretize. Although this method will increase the amount of calculations, it can ensure the consistency of the calculation formula and the data, and at the same time introduce spherical convolution. Accumulating thoughts. This is related to our approach using parameterized differential operators, and some work exploits diffusion kernels for efficient machine learning and CNNs. Kondor and Lafferty were among the first to suggest the use of diffusion kernels on graphs. Atwood and Towsley proposed the diffusion convolutional neural network (DCNN) for efficient convolution on graph-structured data. Boscaini et al. generalized classical CNN to non-Euclidean domains by using a set of directional anisotropic diffusion kernels. Cohen & Welling utilize linear combinations of filter banks to obtain equivariant convolution filters. Ruthotto & Haber explored the reparameterization of convolution kernels using parabolic and hyperbolic differentiation of regular grid images.
3.Non-Euclidean Convolutions
Non-Euclidean convolutions perform convolutions on manifolds represented by unstructured grids (i.e. grids). And work on geometric deep learning solves similar problems. Other methods include graph convolution by parameterizing convolution kernels in the spectral domain , thus converting the convolution step into a spectral dot product. Masci et al. utilize geodesic distance-based cross-correlation to perform convolution directly on the manifold, and Maron et al. use an optimal surface parameterization method (seamless torus covering) to parameterize the zero-general shape into a 2D signal in order to use conventional Planar cnn for analysis.
3. Method
1. Parameterized differential operators
A 3×3 kernel with parameters θ: G 3×3 θ can be written as a linear combination of basis kernels, which can be viewed as delta functions with constant offsets:
Summarize
where x and y refer to the spatial coordinates, corresponding to the two spatial dimensions in which the convolution is performed. Due to the linearity of the cross-correlation operator (*), the output feature map can be expressed as a linear combination of input functions cross-correlated with different basis functions.
Define the linear operator Δij as the cross-correlation function with a basis delta function, as follows:
We replace the cross-correlation linear operator Δij with a variable-order differential operator. Similar to linear operators obtained by cross-correlation of basis functions, differential operators are linear and approximate local features. Unlike cross-correlations on manifolds, differential operators on grids can be efficiently computed using finite element basis or discrete external calculus. In the following practical implementation, we choose the identity (I, zeroth-order differential, same as Δ00), the derivatives of two orthogonal space dimensions (∇x, ∇y, first-order differential) and the Laplacian operator (∇ 2. Second-order differential)
This constitutes this part, that is, the variable itself, the first-order derivative of the variable in the second direction, and the second-order derivative in the two directions: The
scientific point of this method is that the first-order derivative of the atmospheric circulation The calculation reflects the conservation of momentum, and the calculation of the second derivative reflects the mechanics in the atmosphere (pressure force, Coriolis force, gravity, friction). For the sphere, the article chose the east-west and north-south directions as the x and y components because the poles naturally determine the direction of the spherical signal. The Laplacian operator on the grid can be discretized using the cotangent formula:
After that, a multi-layer residual network is added to construct a mesh_block:
where N (i) is the node on the adjacent single ring of i, and Ai is the corresponding With respect to the dual surface area of node i, αij and βij are the two corners of the opposite side ij. With this parameterization of the convolution kernel, the parameters can be similarly optimized by backpropagation using standard stochastic optimization routines. x and y correspond to the NS and EW components of the spherical mesh in the article code
2. Generate spherical mesh
Generating a sphere is mainly based on the icosahedral icosphere method. The characteristic of this method is to first generate an icosahedron, then connect the midpoints of the edges of each triangle to generate four new triangles, and then remap these triangles and points to On the surface of the sphere, a uniform shape is maintained.
This part of the C++ code can be written as
SurfaceMesh icosphere(size_t n_subdivisions)
{
auto mesh = icosahedron();
SurfaceSubdivision subdivision(mesh);
for (size_t i = 0; i < n_subdivisions; i++)
{
subdivision.loop(); //分形
project_to_unit_sphere(mesh); //重映射
}
return mesh;
}
This part is mainly supported by libigl.
First, in terms of import packages, this part of the code is mainly provided by libigl. The article code uses pyigl, but in 2020, the libigl team restructured the code and changed it to igl. Unfortunately, the refactored code usage method has greatly deviated from pyigl, so this part needs to refactor the article code or find a way to find the pyigl package and use the python3.5 environment. This environment can be simply deployed via docker:
docker pull gilureta/pyigl
Regarding the code for constructing mesh, we analyze it:
Then look at fractals:
and remapping:
The original icosahedron is constructed from this part of the code:
Other detailed functions are not listed here, they are mainly responsible for providing some assistance in matrix operations. We use the following instructions to generate the mesh from multiple iterations:
import numpy as np
import mesh
icos = mesh.export_spheres([0,1,2,3],'./')
This generates an icosahedral mesh of order 4.
3. Experimental comparison
The article first compares the number of parameters and the efficiency of the parameters:
due to the parameterized convolution kernel, it can be expected that the number of parameters has increased, but due to the high performance achieved by spherical convolution, these increases have greatly improved the efficiency of the parameters. .
In addition, the article also compares different spherical tasks:
It can be seen that the performance achieved is quite good.
The final article conducts experiments and ablation experiments in climate prediction:
The results in this part are rather rough and require further development and research.