E2CNN: General E(2)-Equivariant Steerable CNNs论文解读

论文地址
代码地址

Introduction

---

The equivariance of neural networks under symmetry group actions has in the recent years proven to be a fruitful prior in network design By guaranteeing a desired transformation behavior of convolutional features under transformations of the network input, equivariant networks achieve improved generalization capabilities and sample complexities compared to their non-equivariant counterparts. Due to their great practical relevance, a big pool of rotation- and reflection- equivariant models for planar images has been proposed by now.

近年来，证明在对称群作用下具有等变性的神经网络被证明在网络设计中极富先验 在网络输入变换下，通过保证进行卷积特征所需要变换，相对于非等变网络，等变网络提升了泛化能力和样本复杂度。 由于他们具有很大的实用意义，当前出现了一大批平面图像的旋转等变和镜像对称等变的网络

An important step in this direction is given by the theory of Steerable CNNs which defines a very general notion of equivariant convolutions on homogeneous spaces. The feature spaces of steerable CNNs are thereby defined as spaces of feature fields, characterized by a group representation which determines their transformation behavior under transformations of the input. In order to preserve the specified transformation law of feature spaces,the convolutional kernels are subject to a linear constraint, depending on the corresponding group representations. While this constraint has been solved for specific groups and representations, no general solution strategy has been proposed so far.

旋转等边和镜像对称等变这个方向重要的一步是可控卷积神经网络的提出，它定义的是一个齐次空间上的等变卷积的普遍概念。 可控卷积神经网络的特征空间被定义为：确定其在输入变化下的群表征(group representation)的特征场空间， 为了保存特征空间的特定变换规律，卷积核会受到相应群表征的线性约束，而这种约束可以通过特定群表征进行求解，只是并没有提出一个通解。

作者大概做了什么？

Author further propose a group restriction operation, allowing for network architectures which are decreasingly equivariant with depth. This is useful e.g. for natural images which show low level features like edges in arbitrary orientations but carry a sense of preferred orientation globally. An adaptive level of equivariance accounts for the resulting loss of symmetry in the hierarchy of features. Since the theory of steerable CNNs does not give a preference for any choice of group representation or equivariant nonlinearity, we run an extensive benchmark study, comparing different equivariance groups, representations and nonlinearities. We do so on MNIST 12k, rotated MNIST SO(2) and reflected and rotated MNIST O(2) to investigate the influence of the presence or absence of certain symmetries in the dataset. A drop in replacement of our equivariant convolutional layers is shown to yield significant gains over non-equivariant baselines on CIFAR10, CIFAR100 and STL-10. Author's contributions are of relevance for general steerable CNNs on homogeneous spaces and gauge equivariant CNNs on manifolds since these models obey the same kind of kernel constraints. More specifically, 2-dimensional manifolds, endowed with an orthogonal structure group O(2) (or subgroups thereof), necessitate exactly the kernel constraints solved in this paper. Author's results can therefore readily be transferred to e.g. spherical CNNs or more general models of geometric deep learning.

作者进一步提出了一个群限制操作，允许网络结构随着网络结构加深而降低等变性。 这是很有用的，比如说，特征水平较低的图片，如包含了偏向某一个方向信息的任意方向的边 自适应等变的水平解释了特征层次结构中导致的对称性损失。 由于可控卷积神经网络的理论没有给出群表征或者非线性等变的选择方法，所以作者比较不同的等变群、等变表征和等变非线性进行了研究 作者在MNIST 12k、旋转的MNIST SO(2)（Special Orthogonal）和镜像翻转与旋转的MNIST O(2)（Orthogonal）数据集，以研究数据集中某些对称性的存在或缺失对结果的影响。 在CIFAR10、CIFAR100和STL-10数据集上，少量替换作者的等变卷积层相比于非等变卷积会明显提升效果 作者的贡献对泛化齐次空间和流形的规范等变可控卷积神经网络有重大意义，由于这些模型遵守相同类型的卷积核约束。更特殊的是用正交结构群O(2)编码的二维流形（或者其中的子群）需要论文中求解的核约束。作者的结论因此能便利地转换到球面空间(球谐)或者更多的泛化几何深度学习模型中。

	order$|G|$	$G⩽O(2)$	$({\mathbb{R}}^2,+)⋊G$
orthogonal	-	O(2)	$E(2)\cong ({\mathbb{R}}^2,+)⋊O(2)$
special orthogonal	-	SO(2)	$SE(2)\cong ({\mathbb{R}}^2,+)⋊SO(2)$
cyclic	N	$C_N$	$({\mathbb{R}}^2,+)⋊C_N$
reflection	2	$(\pm 1,\ast )\cong D_1$	$({\mathbb{R}}^2,+)\cong (\pm 1,\ast )$
dihedral	2N	$D_N\cong C_N⋊(\pm 1,\ast )$	$({\mathbb{R}}^2,+)⋊D_N$

---

\qquad

下面的内容需要补充一点知识基础

由于博文内容限制字数，所以补充的知识点放在了这里“补充知识”，有基础的同学直接略过。

General E(2)-Equivariant Steerable CNNs

The convolutional weight sharing ensures the inference to be translation-equivariant which means that a translated input signal results in a corresponding translation of the feature maps.

卷积网络权重共享保证了推理中是具有平移等变性的，这就意味着平移后的输入信号会导致特征图会产生响应的平移。

\qquad

Isometries of the Euclidean plane ${\mathbb{R}}^2$

$\bullet$ The Euclidean group E(2) is the group of isometries of the plane ${\mathbb{R}}^2$, consisting of translations, rotations and reflections.

$\circ$ 欧几里得群E(2)是\mathbb{R}^2等距(等度量)群，是由平移、旋转和反射构成的。

$\bullet$ Characteristic patterns in images often occur at arbitrary positions and in arbitrary orientations.

$\circ$ 图片的特征通常出现在任意位置和任意方向

$\bullet$ The Euclidean group therefore models an important factor of variation of image features.

$\circ$ 欧几里得群模型化了一个图片特征变量的重要因素

$\bullet$ Euclidean group can be constructed from the translation group $({\mathbb{R}}^2,+)$ and the orthogonal group O ( 2 ) = { O ∈ R 2 × 2 ∣ O T O = i d 2 × 2 } O(2) = \{O \in \mathbb{R}^{2 \times 2}| O^TO=id_{2 \times 2}\} O(2)={ O∈R2×2∣OTO=id2×2​} via the semidirect product operation as $E(2)\cong ({\mathbb{R}}^2,+)⋊O(2)$

$\circ$ 可以将欧几里得群视为由平移群 $({\mathbb{R}}^2,+)$和正交群 O ( 2 ) = { O ∈ R 2 × 2 ∣ O T O = i d 2 × 2 } O(2) = \{O \in \mathbb{R}^{2 \times 2}| O^TO=id_{2 \times 2}\} O(2)={ O∈R2×2∣OTO=id2×2​}通过半直积的方法构成的群

$$E(2)\cong ({\mathbb{R}}^2,+)⋊O(2)$$

其中

$({\mathbb{R}}^2,+)$是平移群。

$O(2)$是正交群。

• The orthogonal group contains all operations leaving the origin invariant, e.g. continuous rotation and reflections.
  
• 正交群包含了所有不变性操作，例如持续的旋转和镜像。
  

$⋊$是半直积，作用就是把两个群合成一个群。

\qquad

$\bullet$ In order to allow for different levels of equivariance and to cover a wide spectrum of related work we consider subgroups of the Euclidean group of the form $({\mathbb{R}}^2,+)⋊G$, defined by subgroups $G⩽O(2).$

$\circ$ 为了可以允许不同等级的不变性和覆盖更多相关工作，作者用$G⩽O(2)$ 定义形如 $({\mathbb{R}}^2,+)⋊G$的欧几里得子群。

$\bullet$ $G$ could be either the special orthogonal group $SO(2)$, the group ( { ± 1 } , ∗ ) (\{±1\}, ∗) ({ ±1},∗) of the reflections along a given axis, the cyclic groups $C_N$, the dihedral groups $D_N$ or the orthogonal group $O(2)$ itself.

$\circ$ $G$可以是特别的正交群$SO(2)$、沿给定轴镜像对称的 ( { ± 1 } , ∗ ) (\{±1\}, ∗) ({ ±1},∗)群、圆柱群$C_N$、二面体群$D_N$或者正交群本身$O(2)$

• $SO(2)$表示无镜像对称的连续旋转
  
• $D_N和C_N$表示包含$N$次$\frac{2\pi }{N}$旋转和镜像(镜像针对D_N)，因此$C_N和D_N$是序数分别为$N和2N(因为存在镜像，所以旋转多一倍)$的离散子群。

\qquad

E(2)-steerable feature fields

$\bullet$ Steerable CNNs define feature spaces as spaces of $steerablefeaturefieldsf:{\mathbb{R}}^2\to {\mathbb{R}}^c$ which associate a $c-dimensional$ feature vector $f(x)\in {\mathbb{R}}^c$ to each point x of a base space, in our case the
plane ${\mathbb{R}}^2$ .

$\circ$ 可控CNNs把特征空间定义为 ：可控特征场：$f:{\mathbb{R}}^2\to {\mathbb{R}}^c$，该可控特征场被c-维特征向量$f(x)\in {\mathbb{R}}^c$关联到基空间的点上，即${\mathbb{R}}^2$ 空间中

$\bullet$ In contrast to vanilla CNNs, the feature fields of steerable CNNs are associated with a transformation law which specifies their transformation under actions of E(2) (or subgroups) and therefore endows features with a notion of orientation

$\circ$ 和原始CNNs(Vanilla CNNs)相比，可控CNNs的特征场关联了特定E(2)或子群的运动法则，因此给特征赋予了方向的概念。

• 图中左边，是一个标量特征场的变换，欧几里得群作用于标量场，只是将像素点从原来的位置移动那个到新的位置，就是$s(x)\mapsto s((tg{)}^{-1}x)=s(g^{-1}(x-t))$，其中的$tg\in ({\mathbb{R}}^2,+)⋊G$，例如光流估计或者梯度图。
• 图中右边，是一个矢量场的变换，$v(x)\mapsto g\cdot v(g^{-1}(x-t))$，对比标量场，矢量场不仅把像素的移动到了新位置，还根据$g\in G$改变了矢量的方向
  \qquad
  
  
  

$\rho$作为图片中变换法则，群表示为：$\rho :G\mapsto GL({\mathbb{R}}^c)$，详细表明了$c$个$channels$的特征向量$f(x)$经过各种变换后，是如何混合到通道信息中的。群表示满足$\rho (g\tilde{g})$的形式，因此可以将群乘法$g\tilde{g}$表示为$c\times c$维度的$\rho (g)$和$\rho (\tilde{g})$矩阵的乘积。

\qquad

$\bullet$ a $\rho$-field transforms under the induced representation$[In{d}_G^{({\mathbb{R}}^2,+)⋊G}\rho ]$ of $({\mathbb{R}}^2,+)⋊G$

$\circ$ 诱导表示为$[In{d}_G^{({\mathbb{R}}^2,+)⋊G}\rho ]$下的$\rho$-field变换$({\mathbb{R}}^2,+)⋊G$为：

\qquad
$$f(x)\mapsto ([In{d}_G^{({\mathbb{R}}^2,+)⋊G}\rho ](tg)\cdot f(x):=\rho (g)\cdot f(g^{-1}(x-t))\text{\hspace{1em}(1)}$$
\qquad

\qquad

\qquad

E(2)-steerable convolutions

\qquad

$\bullet$ In order to preserve the transformation law of steerable feature spaces, each network layer is required to be equivariant under the group actions. The most general equivariant linear map between steerable feature spaces, transforming under ${\rho }_{in}$ and ${\rho }_{out}$ , is given by $convolutions$ with $G-steerablekernel{s}^{k}k:{\mathbb{R}}^2\to {\mathbb{R}}^{c_{out}\times c_{in}}$，satisfying a kernel constraint:

$\circ$ 为了保留可控制特征空间的变换规律，需要每个网络层在群作用下是等变的。经过${\rho }_{in}$ and ${\rho }_{out}$变换的可控特征空间之间的普遍等变的显性映射是由$G$-$steerable$卷积核${}^k$给定的，$k$个卷积核表示为$kernel{s}^{k}k:{\mathbb{R}}^2\to {\mathbb{R}}^{c_{out}\times c_{in}}$ ，满足如下的核约束：

\qquad
$$k(gx)={\rho }_{out}(g)k(x){\rho }_{in}(g^{-1})\forall g\in G,x\in {\mathbb{R}}^2\text{\hspace{1em}(2)}$$
\qquad

Intuitively,就非变换的坐标$x$的卷积核和卷积核对输入场的变换响应而言，这种约束决定了变换后的坐标$gx$的卷积核的形式。其确保了当输入场根据$In{d}_{{\rho }_{in}}$变换时，输出特征场会变换为$In{d}_{{\rho }_{out}}$。

\qquad

\qquad

由于核约束是线性的，其解会形成一个传统CNNs中非约束的卷积核的矢量空间的线性子空间.

\qquad

\qquad

Irrep(irreducible representation不可约表示,诱导表示一般可约) decomposition of the kernel constraint

\qquad

\qquad

如果任何维数大于一的表示的所有矩阵都可以用相同的相似变换转换为相同的块对角矩阵结构，则称此表示为可约表示，反之称为不可约表示。

\qquad

\qquad

$\bullet$ The kernel constraint (2) in principle needs to be solved individually for each pair of input and output types ${\rho }_{in}$ and ${\rho }_{out}$ to be used in the network.

$\circ$ 原则上对于如下网络中使用的每个输入输出对的类型${\rho }_{in}$ 和 ${\rho }_{out}$来说，需要单独求解上述核约束

• 网络：
  
  $\bullet$A numerical solution technique which is based on a Clebsch-Gordan decomposition of tensor products of irreps has been proposed
  
  $\circ$一种基于不可约的张量积的Clebsch-Gordan分解(又称向量耦合分级)
  

\qquad

• 核约束
  

\qquad
$$k(gx)={\rho }_{out}(g)k(x){\rho }_{in}(g^{-1})\forall g\in G,x\in {\mathbb{R}}^2\text{\hspace{1em}(2)}$$
\qquad

$\bullet$ Any representation of a finite or compact group decomposes under a change of basis into a direct sum of irreps, each corresponding to an invariant subspace of the representation space ${\mathbb{R}}^c$ on which $\rho$ acts. Denoting the change of basis by $Q$, this means that one can always write $\rho =Q^{-1}\left[{⨁}_{i\in I}{\psi }_i\right]Q$ where ${\psi }_i$ are the irreducible representations of $G$ and the index set $I$ encodes the types and multiplicities of irreps present in $\rho$ .

$\circ$ 有限群或紧群的任何表示在基改变的作用下分解为不可约直和(direct sum of irreps)，每个不可约直和对应于经过$\rho$变换作用后的${\mathbb{R}}^c$表示空间的不变子空间。基根据$Q$变换，这意味着，总有$\rho =Q^{-1}\left[{⨁}_{i\in I}{\psi }_i\right]Q$，其中${\psi }_i$ 是$G$的不可约表示，且索引集合$I$编码了$\rho$的不可约表达的类型和稳态解（multiplicity）。

The decomposition of ${\rho }_{in}$ and ${\rho }_{out}$ in the kernel constraint leads to：
核约束中的${\rho }_{in}$ 和 ${\rho }_{out}$的分解会产生：

\qquad

$$\begin{array}{lllclc}k(gx)& =Q_{out}^{-1}& \left[⨁{}_{i\in I_{out}}{\psi }_i(g)\right]& Q_{out}k(x)Q_{in}^{-1}& \left[⨁{}_{j\in I_{in}}{\psi }_j^{-1}(g)\right]Q_{in}& \forall g\in G,x\in {\mathbb{R}}^2\\ & & & & & \\ k(gx)& =& \left[⨁{}_{i\in I_{out}}{\psi }_i(g)\right]& \kappa & \left[⨁{}_{j\in I_{in}}{\psi }_j^{-1}(g)\right]& \forall g\in G,x\in {\mathbb{R}}^2\end{array}$$
其中：

$\bullet$ 将与不可约基相关的核k定义为$\kappa :=Q_{out}k{Q}_{in}^{-1}$

\qquad

$\bullet ⨁$是直和

$V_1$和$V_2$是域$\mathbb{F}$向量空间的两个向量，满足：

V 1 ∩ V 2 = { 0 } V_1 \cap V_2 = \{0\} V1​∩V2​={ 0}
且任意$v\in V$都能$V_1$和$V_2$中向量的线性组合，即
$$v=c_1{v}_1+c_2{v}_2(v1\in V_1,v_2\in V_2,c_1,c_2\in \mathbb{F})$$
那么空间 $V$ 就是$V_1$和$V_2$的直和空间，用直和（direct sum）运算记为
$$V=V_1⨁V_2$$
即 “任意 $v\in V$ 都能表示为$V_1$和$V_2$中向量之和”

$\bullet$ 左右不可约直和的乘积部分可替换为如下形式：

\qquad
$${\kappa }^{ij}(gx)={\psi }_i(g){\kappa }^{ij}(x){\psi }_j^{-1}(g)\forall g\in G,x\in {\mathbb{R}}^{2}whereiin{I}_{out},j\in I_{in}\text{\hspace{1em}(3)}$$
k 1 , ⋯   , k d   : =   ⋃ i ∈ I o u t ⋃ j ∈ I i n   { Q o u t − 1   k ˉ 1 i j   Q i n } (4) \qquad \\ {k_1, \cdots, k_d} \ := \ \bigcup{_{i \in I_{out}}} \bigcup{_{j \in I_{in}}} \ \{Q^{-1}_{out} \ \bar{k}^{ij}_{1} \ Q_{in}\} \tag{4} k1​,⋯,kd​ := ⋃i∈Iout​​⋃j∈Iin​​ { Qout−1​ kˉ1ij​ Qin​}(4)
其中

{ k 1 i j , ⋯   , k d i j i j } \{k^{ij}_{1}, \cdots, k^{ij}_{d_{ij}}\} { k1ij​,⋯,kdij​ij​}是一个$d_{ij}$维的基

$d={\sum }_{ij}{d}_{ij}$，视为$i\times j$个不同的$d$相加
$\bar{k}$指的是在$k$形的矩阵被填充在矩阵块$k^{ij}$的对应位置，而该矩阵块其他位置全都是0.

\qquad

\qquad

General solution of the kernel constraint for O(2) and subgroups

\qquad

$\bullet$In order to build isometry-equivariant CNNs on ${\mathbb{R}}^2$ we need to solve the irrep constraints (3) for the specific case of $G$ being $O(2)$ or one of its subgroups. For this purpose note that the action of $G$ on ${\mathbb{R}}^2$ is norm-preserving, that is, $|g\cdot x|=|x|\forall g\in G,x\in {\mathbb{R}}^2$ . The constraints (2) and (3) therefore only restrict the angular parts of the kernels but leave their radial parts free. It is convenient to expand the kernel w.l.o.g. in terms of an (angular) Fourier series

$\circ$为了在${\mathbb{R}}^2$上建立度量等变的CNNs，我们需要解正交群O2或者其子群下的不可约约束。为此，${\mathbb{R}}^2$上的G： $|g\cdot x|=|x|\forall g\in G,x\in {\mathbb{R}}^2$ 。约束2和3因此只限制了核的角部分，而径向部分没有约束。不是一般性(w.l.o.g.)，很容易把公式转换成傅里叶级数的形式

$${\kappa }_{\alpha \beta }^{ij}(x(r,\varphi ))=A_{\alpha \beta ,0}(r)+\sum _{\mu =1}^{\infty }\left[A_{\alpha \beta ,\mu }(r)\cos (\mu \varphi )+B_{\alpha \beta ,\mu }(r)\sin (\mu \varphi )\right]\text{\hspace{1em}(5)}$$
\qquad

对于每个矩阵块${\kappa }^{ij}$的${\kappa }_{\alpha \beta }^{ij}$的实值的径向相关系数(real-value, radically dependent coefficients)即正弦分量的系数和余弦分量的系数为

$$\begin{array}{clc}{A}_{\alpha \beta ,\mu }& :& {\mathbb{R}}^{+}\to \mathbb{R}\\ B_{\alpha \beta ,\mu }& :& {\mathbb{R}}^{+}\to \mathbb{R}\end{array}$$
把${\kappa }_{\alpha \beta }^{ij}$插入不可约的约束(3)${\kappa }^{ij}(gx)={\psi }_i(g){\kappa }^{ij}(x){\psi }_j^{-1}(g)$中，且投影到单谐（一个频率的谐函数）中，我们可以得到傅里叶系数的约束。

\qquad

---

${\psi }_i$ \ ${\psi }_j$	trivial	sign-flip	frequency $n\in {\mathbb{N}}^{+}$
trivial	$[1]$	$\varnothing$	$[sin(n\varphi ),-cos(n\varphi )]$
sign-flip	$\varnothing$	$[1]$	$[cos(n\varphi ),sin(n\varphi )]$
frequency $m\in {\mathbb{N}}^{+}$	$\left[\begin{array}{c}sin(m\varphi )\\ -cos(m\varphi )\end{array}\right]$	$\left[\begin{array}{c}cos(m\varphi )\\ sin(m\varphi )\end{array}\right]$	$\left[\begin{array}{cc}cos((m-n)\varphi )& -sin((m-n)\varphi )\\ sin((m-n)\varphi )& cos((m-n)\varphi )\end{array}\right]$,$\left[\begin{array}{cc}cos((m+n)\varphi )& sin((m+n)\varphi )\\ sin((m+n)\varphi )& -cos((m+n)\varphi )\end{array}\right]$

O(2)可控核的角部分的基，满足不同不可约输入场${\psi }_i$和不可约输出${\psi }_j$场对的不可约约束(3)

\qquad
\qquad

Group representation and nonlinearities

\qquad

$\bullet$ Considering only the convolution operation with $G-steerable$ kernels for the moment, it turns out that any change of basis $P$ to an equivalent representation $\tilde{\rho }:=P^{-1}\rho P$ is irrelevant

$\circ$ 考虑到运动中只有$G-steerable$卷积核的操作，对于一个等变表示为$\tilde{\rho }:=P^{-1}\rho P$ 的任何$P$基的变换是相关的

\qquad

$\bullet$ Consider the irrep decomposition$\rho =Q^{-1}\left[⨁{}_{i\in I}{\psi }_i\right]Q$used in the solution of the kernel constraint to obtain a basis { k i } i = 1 d \{k_i\}^{d}_{i=1} { ki​}i=1d​ of G-steerable kernels as defined by Eq. (4). Any equivalent representation will decompose into $\tilde{\rho }={\tilde{Q}}^{-1}\left[⨁{}_{i\in I}{\psi }_i\right]Q$ with $\tilde{Q}=QP$ for some $P$ and therefore result in a kernel basis { P o u t − 1 k i P i n } i = 1 d \{P^{-1}_{out}k_i P_{in}\}^{d}_{i=1} { Pout−1​ki​Pin​}i=1d​ which entirely negates changes of bases between
equivalent representations. It would therefore w.l.o.g. suffice to consider direct sums of irreps$\rho =⨁{}_{i\in I}{\psi }_i$ as representations only

$\circ$ 考虑等式(4)中用于获取G-steerable卷积核约束解的不可约分解$\rho =Q^{-1}\left[⨁{}_{i\in I}{\psi }_i\right]Q$。任何等变表示都可以用$\tilde{Q}=QP$分解为$\tilde{\rho }={\tilde{Q}}^{-1}\left[⨁{}_{i\in I}{\psi }_i\right]Q$，并且产生卷积核的基底 { P o u t − 1 ， k i P i n } i = 1 d \{P^{-1}_{out}，k_i P_{in}\}^{d}_{i=1} { Pout−1​，ki​Pin​}i=1d​，该基底让等变表示的基变化完全无用。不是一般性，这足以视不可约直和$\rho =⨁{}_{i\in I}{\psi }_i$为仅有的表示。

\qquad

$\bullet$ In practice, however, convolution layers are interleaved with other operations which are sensitive to specific choices of representations. In particular, nonlinearity layers are required to be equivariant under the action of specific representations. The choice of group representations in steerable CNNs therefore restricts the range of admissible nonlinearities, or, conversely, a choice of nonlinearity allows only for certain representations.

$\circ$ 然而，实际上各卷基层和其他特定特征选择的操作是交织在一起的。尤其是，在特定表示作用下，非线性层需要是等变的。在可控卷积网络中，群表示的选择限制了一系列允许的线性变换，或者，相反地，非线性选择仅仅适用于某些表达

\qquad

$\bullet$ All equivariant nonlinearities considered here act spatially localized, that is, on each feature vector $f(x)\in {\mathbb{R}}^{c_{in}}$ for all $x\in {\mathbb{R}}^2$ individually. They might produce different types of output fields ${\rho }_{out}:G\to GL({\mathbb{R}}^{c_{out}})$, that is, $\sigma :{\mathbb{R}}^{c_{in}}\to {\mathbb{R}}^{c_{out}},f(x)\mapsto \sigma (f(x))$. It is sufficient to require the equivariance of $\sigma$ under the actions of ${\rho }_{in}$ and ${\rho }_{out}$, i.e. $\sigma \circ {\rho }_{in}(g)={\rho }_g\circ \sigma \forall g\in G$, for the nonlinearities to be equivariant under the action of induced representations when being applied to a whole feature field as $\sigma (f)(x):=\sigma (f(x))$.

$\circ$ 这里所考虑的所有等变非线性操作用于空间($H\times W$空间)定位

，即在所有位于空间$x\in {\mathbb{R}}^2$的每个特征向量$f(x)\in {\mathbb{R}}^{c_{in}}$。这会产生不同输入场${\rho }_{out}:G\to GL({\mathbb{R}}^{c_{out}})$的类型，即 $\sigma :{\mathbb{R}}^{c_{in}}\to {\mathbb{R}}^{c_{out}},f(x)\mapsto \sigma (f(x))$。在${\rho }_{in}$ 和 ${\rho }_{out}$作用下，仅需要等变性$\sigma$就足够了，例如，$\sigma \circ {\rho }_{in}(g)={\rho }_g\circ \sigma \forall g\in G$，当用于整个特征场$\sigma (f)(x):=\sigma (f(x))$，对于在诱导表示(可约)作用下，非线性操作是等变的

\qquad

$\bullet$A general class of representations are $unitaryrepresentations$ which preserve the norm of their representation space, that is, they satisfy $|{\rho }_{unitary}(g)f(x)|=|f(x)|\forall g\in G$

$\circ$ 一个通用类的表示是酉表示(unitary representation)，它保存了表示空间的范数，这满足$|{\rho }_{unitary}(g)f(x)|=|f(x)|\forall g\in G$

什么是酉表示？



酉表示，群到酉群的同态，是表示论中 一种经典群表示方法，酉表示论的主要目标之一是描述“酉对偶”，即 G 的所有不可约酉表示的空间。

旨在将抽象代数结构中的元素“表示”成向量空间上的线性变换，并研究这些代数结构上的模，藉以研究结构的性质。

略言之，表示论将一代数对象表作较具体的矩阵，并使得原结构中的代数运算对应到矩阵加法和矩阵乘法。

设$G$为群，其在域$F$（常取复数域$F=\mathbb{C}$）表示是: $F$矢量空间$V$及映至一般线性群之群同态

$$\rho :G\to GL(V)$$
假设$V$有限维，则上述同态即是将$G$的元素映成可逆矩阵，并使得群运算对应到矩阵乘法。


表示论的妙用在于能将抽象代数问题转为较容易解决的线性代数问题。此外，群还可以表示在无穷维空间上；

表示论的一大特点是它遍布数学各个领域。这个特点有两个方面。首先，表示论的应用十分广泛：除了在代数的影响之外，表示论

• 通过调和分析阐明并推广了傅里叶分析；
• 通过不变量理论和爱尔兰根纲领与几何学建立了联系；
• 通过自守形式和朗兰兹纲领对数论产生了影响。

\qquad

$\bullet$ Nonlinearities which solely act on the norm of feature vectors but preserve their orientation are equivariant w.r.t. unitary representations. They can in general be decomposed in ${\sigma }_{norm}:{\mathbb{R}}^c\to {\mathbb{R}}^c,f(x)\mapsto \eta (|f(x)|\frac{f(x)}{|f(x)|}$ for some nonlinear function $\eta :{\mathbb{R}}_{⩾0}\to {\mathbb{R}}_{⩾0}$ acting on the norm of feature vectors.

$\circ$ 仅仅作用于特征向量范数但保存了特征向量方向的非线性操作是关于酉表示等变的。他们通常可以分解为 ${\sigma }_{norm}:{\mathbb{R}}^c\to {\mathbb{R}}^c,f(x)\mapsto \eta (|f(x)|\frac{f(x)}{|f(x)|}$ ，在对某些作用于特征向量的范数的非线性操作下：$\eta :{\mathbb{R}}_{⩾0}\to {\mathbb{R}}_{⩾0}$

\qquad

$\bullet$A common choice of representations of finite groups like $C_N$ and $D_N$ are regular representations. Their representation space ${\mathbb{R}}^{|G|}$ has dimensionality equal to the order of the group, e.g. ${\mathbb{R}}^N$ for $C_N$ and ${\mathbb{R}}^{2N}$ for $D_N$. The action of the regular representation is defined by assigning each axis $e_g$ of ${\mathbb{R}}^{|G|}$ to a group element $g\in G$ and permuting the axes according to ${\rho }_{reg}^G(\tilde{g})e_g:=e_{\tilde{g}g}$ . Since this action is just permuting channels of ${\rho }_{reg}^G$-fields, it commutes with pointwise nonlinearities like ReLU.

$\circ$ 像$C_N$和$D_N$有限群表示的一个通用的选择是使用三角表示。三角表示空间${\mathbb{R}}^{|G|}$ 和群相同的阶数(order)，例如 ${\mathbb{R}}^N$ 对于 $C_N$ 和 ${\mathbb{R}}^{2N}$ 对于 $D_N$，都是具有相同阶数的。三角表示的作用是通过把${\mathbb{R}}^{|G|}$d每个轴$e_g$分给群元素$g\in G$并且根据${\rho }_{reg}^G(\tilde{g})e_g:=e_{\tilde{g}g}$对各个轴进行排序。由于这种变换只是对${\rho }_{reg}^G$场的通道进行排序，所以他是和点级的非线性操作进行交互的，例如ReLU。

\qquad

$\bullet$ While regular steerable CNNs were empirically found to perform very well, they lead to high dimensional feature spaces with each individual field consuming $|G|$ channels. The translation of feature maps of conventional CNNs can be viewed as action of the regular representation of the translation group.

$\circ$ 尽管三角可控卷积网络表现很好，但是会导致高维特征空间的每个单独的场都会占据一个$|G|$通道。 卷积网络的特征图的平移会被视为平移群的三角表示的作用。

\qquad

$\bullet$ Closely related to regular representations are quotient representations. Instead of permuting $|G|$ channels indexed by $G$, they permute $|G|/|H|$ channels indexed by cosets $gH$ in the quotient
space $G/H$ of a subgroup $H⩽G$. Specially，they act on axes $e_{gH}$ of ${\mathbb{R}}^{|G|/|H|}$ as defined by ${\rho }_{quot}^{G/H}(\tilde{g})e_{g}H:=e_{\tilde{g}gH}$. This definition covers regular representations as a special case for the trivial subgroup H = { e } H = \{e\} H={ e}.

$\circ$ 商表示是一种非常接近三角表示的方式。 商表示使用在子集 $H⩽G$ 的商空间 $G/H$ 的陪集 $gH$ 来对 $|G|/|H|$ 进行排序，而非通过 $G$ 对 $|G|$ 通道进行排序和索引。具体来说就是，作用于${\mathbb{R}}^{|G|/|H|}$空间的轴$e_{gH}$被定义为 ${\rho }_{quot}^{G/H}(\tilde{g})e_{g}H:=e_{\tilde{g}gH}$。 这种定义覆盖了三角标示，是一种平凡子群 H = { e } H = \{e\} H={ e}的特殊情况

什么是平凡子群？

\qquad

任意群$G$，都含有 { e } ( e \{e\} (e { e}(e是单位圆$)$ 和$G$作为子群的，称为平凡子群。

什么是陪集群coset group？

\qquad

陪集是:

$H$是$G$的子群(左图)，右图中，在$G$中选一个不属于$H$的元素$g$，将其乘以$H$得到$gH$ ，注意若非阿贝尔群，$gH$和$Hg$是不同的，称$gH$为左陪集(Left Coset)，可以证明$H$和$gH$无交集不重叠；类似的，在$G$中选择一个不属于$H$和$gH$的元素KaTeX parse error: Expected group after '^' at position 2: g^̲'乘以$H$得到KaTeX parse error: Expected group after '^' at position 2: g^̲'H，与前两者都不重叠，…… ，这样一直进行下去，通过选择大小相同但不重叠的陪集，完成对$G$的划分($G$是有限群)。


陪集不是子群，因为它没有恒等元$e$，而$H$有。

陪集并不总能形成一个群。陪集能组成群的充要条件是对于任何 $y\in G$ ，都能满足 $y^{-1}Ny=N$ ，$N$ 是一个正规子群，记 $N⊴G$ 。

陪集群称为因子群(factor group)，记$G/N$，在该群中，恒等元为$N$

\qquad

$\bullet$ Both regular and quotient representations can be viewed as being induced from the trivial representation of a subgroup $H⩽G$, specifically ${\rho }_{reg}^G=In{d}_{quot}^{G}1$ and ${\rho }_{quot}^G=In{d}_H^{G}1$. More generally, any representation$\tilde{\rho }:=H\to GL({\mathbb{R}}^c)$can be used to define an induced representations ${\rho }_{ind}=In{d}_H^G\tilde{\rho }:G\to GL({\mathbb{R}}^{c\cdot |G:H|})$. Here $|G:H|$ denot the index of $H$ in $G$ which corresponds to $|G|/|H|$ if $G$ and $H$ are both finite.

$\circ$ 三角标示和商表示都能被视为$H⩽G$子群的平凡表示(仅有$e$)的约分，详细说就是 ${\rho }_{reg}^G=In{d}_{quot}^{G}1$ 和 ${\rho }_{quot}^G=In{d}_H^{G}1$。更通俗地说，任何表示$\tilde{\rho }:=H\to GL({\mathbb{R}}^c)$都可以用于定义一个可约表示${\rho }_{ind}=In{d}_H^G\tilde{\rho }:G\to GL({\mathbb{R}}^{c\cdot |G:H|})$。其中，$|G:H|$指的是$|G|$和$|H|$都是有限群时，$|G|/|H|$所对应的$G$的指数index

\qquad

$\bullet$ Regular and quotient fields can furthermore be acted on by nonlinear pooling operators. Via a group pooling or projection operation max :${\mathbb{R}}^c\to \mathbb{R},f(x)\to max(f(x))$ the works extract the maximum value of a regular or quotient field. The invariance of the maximum operation implies that the resulting features form scalar fields. Since group pooling operations discard information on the feature orientations entirely, vector field nonlinearities ${\sigma }_{vect}:{\mathbb{R}}^N\to {\mathbb{R}}^2$ for regular representations of $C_N$ were proposed in other paper. Vector field nonlinearities do not only keep the maximum response max$(f(x))$ but also its index arg max$(f(x))$. This index corresponds to a rotation angle $\theta =\frac{2\pi }{N}argmax(f(x))$ which is used to define a vector field with elements $v(x)=max(f(x))(cos(\theta ),sin(\theta ){)}^T$.

$\circ$ 三角场和商场可以作用于非线性池化中。 通过group pooling或者最大投影操作: ${\mathbb{R}}^c\to \mathbb{R},f(x)\to max(f(x))$，提取正规或者商场的最大值。最大值操作的不变性意味着生成的特征是标量场。由于group pooling操作会完全放弃特征的方向信息，所以，其他论文中提出了$C_N$的正规表示的非线性向量场${\sigma }_{vect}:{\mathbb{R}}^N\to {\mathbb{R}}^2$。非线性向量场不经保存了$(f(x))$的最大响应，还保存了指数index的$argmax(f(x))$。该指数对应了旋转角$\theta =\frac{2\pi }{N}argmax(f(x))$，此角度是用元素 $v(x)=max(f(x))(cos(\theta ),sin(\theta ){)}^T$ 定义矢量场

\qquad

$\bullet$ In general, any pair of feature fields $f_1:{\mathbb{R}}^2\to {\mathbb{R}}^{c_1}$ and $f_2:{\mathbb{R}}^2\to {\mathbb{R}}^{c_2}$ can be combined via the tensor product operation $f_1\otimes f_2$. Given that the individual fields transform under arbitrary representations ${\rho }_1$ and ${\rho }_2$ , their product transforms under the tensor product representation ${\rho }_1\otimes {\rho }_2$. Any pair $f_1$ and $f_2$ of feature fields can furthermore be concatenated by taking their direct sum $f_1\oplus f_2:{\mathbb{R}}^2\to {\mathbb{R}}^{c_1+c_2}$ which we used in equation(1) to define feature spaces comprising multiple feature fields. The concatenated field transforms according to the direct sum representation as $({\rho }_1\oplus {\rho }_2)(g)(f_1\oplus f_2):={\rho }_1(g)f_1\oplus {\rho }_2(g)f_2$

$\circ$ 通常，任何特征场对$f_1:{\mathbb{R}}^2\to {\mathbb{R}}^{c_1}$和$f_2:{\mathbb{R}}^2\to {\mathbb{R}}^{c_2}$ 能通过卷积操作$f_1\otimes f_2$联系在一起。如果在任意表示${\rho }_1$ 和 ${\rho }_2$作用下进行场变换，那么他们的积会随着张量积表示${\rho }_1\otimes {\rho }_2$而变换。任何特征场对$f_1$和$f_2$可以通过其直和$f_1\oplus f_2:{\mathbb{R}}^2\to {\mathbb{R}}^{c_1+c_2}$进行拼接，这个是用于公式(1)的特征空间定义，包含了多重特征场。拼接后的特征场会根据直和变换，如$({\rho }_1\oplus {\rho }_2)(g)(f_1\oplus f_2):={\rho }_1(g)f_1\oplus {\rho }_2(g)f_2$，相当于做了个分解。

\qquad

\qquad

Group restrictions and inductions

\qquad

$\bullet$ The key idea of equivariant networks is to exploit symmetries in the distribution of characteristic patterns in signals. The level of symmetry present in data might thereby vary over different length scales. For instance, natural images typically show small features like edges or intensity gradients in arbitrary orientations and reflections. On a larger length scale, however, the rotational symmetry is broken as manifested in visual patterns exclusively appearing upright but still in different reflections. Each individual layer of a convolutional network should therefore be adapted to the symmetries present in the length scale of its fields of view.

$\circ$ 等变网络的关键思想是采取信号中的特征模式分布的对称性。出现在数据中的对称性等级会在长度尺度上变化。例如，自然图像通常在任意方向和和镜像对称中会展示一些小特征，比如边和梯度强度等。然而在更大的长度尺度上，旋转的对称性会被打破，表现为视觉特征出现竖直但仍然在不同的镜像反射中。因此，卷积网络的每个单独的层都会适应这种对称性存在于其视场的长度尺度中。

\qquad

$\bullet$ A loss of symmetry can be implemented by restricting the equivariance constraints at a certain depth to a subgroup $({\mathbb{R}}^2,+)⋊H⩽({\mathbb{R}}^2,+)⋊G$, where $H⩽G$; e.g. from rotations and reflections $G=O(2)$ to mere reflections H = ( { ± 1 } , ∗ ) H=(\{\pm 1 \}, *) H=({ ±1},∗) in the example above. This requires the feature fields produced by a layer with a higher level of equivariance to be reinterpretered in the following layer as fields transforming under a subgroup. Specially, a $\rho$-field, transforming under a representation $\rho \to G\to GL({\mathbb{R}}^c)$, needs to be reinterpreted as a $\tilde{\rho }$-field, where $\tilde{\rho }:H\to GL({\mathbb{R}}^c)$ is a representation of the subgroup $H⩽G$. This is naturally achieved by define $\tilde{\rho }$ to be restricted representation:

$\circ$ 可以通过限制某些深度的等变约束到子群$({\mathbb{R}}^2,+)⋊H⩽({\mathbb{R}}^2,+)⋊G$上，实现对称损失，该子群 中$H⩽G$，例如从旋转和镜像反射$G=O(2)$到只有镜像反射 H = ( { ± 1 } , ∗ ) H=(\{\pm 1 \}, *) H=({ ±1},∗)。这就要求由较高等变性水平的层生成的特征场在随后的层中重新解释为子群作用下变换的特征场。尤其是，在$\rho :G\to GL({\mathbb{R}}^c)$表示下变换的$\rho$场需要被重新解释为一个$\tilde{\rho }$场，即$\tilde{\rho }:H\to GL({\mathbb{R}}^c)$是子群$H⩽G$的表示。通过定义$\tilde{\rho }$为受限表达：

\qquad

$$\tilde{\rho }:=Re{s}_H^G(\rho ):H\to GL({\mathbb{R}}^c),h\mapsto \rho (h)$$
\qquad

\qquad

Implementation details

Author’s implemetation involves:

• 计算可控卷积核的偏置
• 具有拓展系数的上述偏置下，可控卷积核的拓展(expansion)
• 实际卷积路径的进行

\qquad

公式1

$$f(x)\mapsto ([In{d}_G^{({\mathbb{R}}^2,+)⋊G}\rho ](tg)\cdot f(x):=\rho (g)\cdot f(g^{-1}(x-t))\text{\hspace{1em}(1)}$$
\qquad

公式2
$$k(gx)={\rho }_{out}(g)k(x){\rho }_{in}(g^{-1})\forall g\in G,x\in {\mathbb{R}}^2\text{\hspace{1em}(2)}$$
\qquad

公式3
$${\kappa }^{ij}(gx)={\psi }_i(g){\kappa }^{ij}(x){\psi }_j^{-1}(g)\forall g\in G,x\in {\mathbb{R}}^{2}whereiin{I}_{out},j\in I_{in}\text{\hspace{1em}(3)}$$
\qquad

公式4
k 1 , ⋯   , k d   : =   ⋃ i ∈ I o u t ⋃ j ∈ I i n   { Q o u t − 1   k ˉ 1 i j   Q i n } (4) \qquad \\ {k_1, \cdots, k_d} \ := \ \bigcup{_{i \in I_{out}}} \bigcup{_{j \in I_{in}}} \ \{Q^{-1}_{out} \ \bar{k}^{ij}_{1} \ Q_{in}\} \tag {4} k1​,⋯,kd​ := ⋃i∈Iout​​⋃j∈Iin​​ { Qout−1​ kˉ1ij​ Qin​}(4)
\qquad

公式5
$${\kappa }_{\alpha \beta }^{ij}(x(r,\varphi ))=A_{\alpha \beta ,0}(r)+\sum _{\mu =1}^{\infty }\left[A_{\alpha \beta ,\mu }(r)\cos (\mu \varphi )+B_{\alpha \beta ,\mu }(r)\sin (\mu \varphi )\right]\text{\hspace{1em}(5)}$$

给定$G⩽O(2)$下的输入${\rho }_{in}$和输出${\rho }_{out}$ ，首先预计算一组满足等式(2)的$G$-steerable卷积核的一组偏置 { k 1 , ⋯   , k d } \{k_1, \cdots, k_d\} { k1​,⋯,kd​}。

为了解核约束，作者计算了不可约输入输出的表示的类型和稳态解。

通过解线性系统 $\rho (g)=Q^{-1}\left[⨁{}_{i\in I}{\psi }_i(g)\right]Q\forall g\in G$ 可以获得偏置的改变。

代码在e2cnn/group/representation.py中求直和

cob = np.zeros((size, size))
cob_inv = np.zeros((size, size))
p=0
for r in reqrs:
	# 把矩阵放在对角线上 
	cob[c] = r.change_of_basis
	cob_inv[p:p + r.size, p:p + r.size] = r.change_of_basis_inv
	p += r.size

对于输入${\rho }_{in}$和输出${\rho }_{out}$ 的每对不可约${\psi }_i$和${\psi }_j$，作者得到解析解 { κ 1 i j , … , κ d i , j i j } \{\kappa^{ij}_1, \dots , \kappa^{ij}_{d_{i, j}}\} { κ1ij​,…,κdi,j​ij​}

同偏置矩阵$Q_{in}$和$Q_{out}$的变化，他们共同决定了G-steerable卷积核的偏置 { k 1 , … , k d } \{k_1, \dots, k_d\} { k1​,…,kd​}的三角表示部分。

由于核空间约束仅仅影响核的三角表示，所以我们可以自由选择任何径向的轮廓。

作者选择了高斯分布的径向轮廓$e^{\frac{1}{2{\sigma }^2}(r-R{)}^2}$，标准差$\sigma$，中心的半径为$R=1,\dots ,└s/2┘$

关于$r$、$R$和$\sigma$的生成
首先在e2cnn/nn/modules/r2_conv/r2convolution.py中的:

def compute_bias_params(kernelsize: int,
						frequency_cutoff: Union[float, Callable[[float], float]] = None,
						rings: List[float] = None,
						sigma: List[float]=None,
						dilation: int = 1,
						custom_basis_filter: Callable[[dict], bool] = None):
	if rings is None:
		n_rings = math.ceil(kernel_size / 2)
		rings = torch.linspace(0, (kernel_size - 1) // 2, n_rings)  * dilation
		rings = rings.tolist()
	if sigma is None:
		# 变成[0.6, 0.6, 0.6, 0.4]
		sigma = [0.6] * (len(rings)-1) + [0.4]
		for i, r in enumerate(rings):
			if r == 0. :
				# 避免出现sigma=0的情况，因为sigma是分母
				sigma[i] = 0.005

> 然后再在e2cnn/kernels/basis.py中，把rings(例如图片中的$rings=[0,1,2,3]$)分别变成

self.radii = np.array(rings).reshape(1, 1, -1, 1) #这就是r
radii = radii.reshape(1, 1, 1, -1) #这就是R
"""对角线为0的矩阵
[[0,  1,  4,  9],
 [1,  0,  1,  4],
 [4,  1,  0,  1],
 [9,  4,  1,  0]]
"""
d = (self.radii - radii) **2
out = np.exp(-0.5*d/sigma**2)

如何使用E2CNN代码？

1. 首先确定输入特征或者图片的输入通道、输出通道和旋转方式

# 旋转8次，每次 pi/4 = 45°
orientation = 8
# self.gspace是包含了所有旋转信息的函数（类似字典），通过属性调用
self.gspace = e2cnn.gspaces.Rot2dOnR2(orientation)
self.in_type = e2cnn.FieldType(self.gspace, [self.gspace.trivial_repr] * input_channels)
self.out_type = e2cnn.FieldType(self.gspace, [self.gspace.trivial_repr] * output_channels)

2. 然后把输入特征或者图片变成Geometric_tensor，这里就是一个封装函数，属性通过.attribute来调用

# 这里是e2cnn专用的Geometric_tensor，如果想变成普通的tensor，只需要x.tensor，就可以让普通卷积使用
x = img # 或者 x = feature_map
x = e2cnn.GeometricTensor(x, self.in_type)

3. 初始化e2cnn卷积函数、ReLU函数、Norm函数和Pool函数

in_type = self.in_type
out_tpye = self.out_type
stride = 1
padding = 0
dilation = 1
bias = False
self.conv_nxn = e2cnn.R2Conv(in_type, 
							 out_type, 
							 n, # n就是kernel_size
							 stride=stride,
							 padding=padding,
							 dilation=dilation,
							 bias=bias,
							 sigma=None,
							 # 这里的frequencies_cutoff必须是可迭代对象
							 frequencies_cutoff=lambda r: 3 * r)
self.relu = e2cnn.ReLU(self.conv_nxn.out_type, inplace=True)
self.norm = e2cnn.InnerBatchNorm(self.conv_nxn.out_type)
self.pool = e2cnn.PointwiseMaxPool(self.conv_nxn.out_type, 
								   kernel_size=n, 
								   stride=stride, 
								   padding=padding)
self.interp = e2nn.R2Upsampling(self.pool.out_type, 
								scale_factor, 
								mode='nearest', 
								align_corners=True)
# 或者使用这个， 二选其一
self.relu = e2cnn.ReLU(self.out_type, inplace=True)
self.norm = e2cnn.InnerBatchNorm(self.out_type)
self.pool = e2cnn.PointwiseMaxPool(self.out_tpye, 
								   kernel_size=n, 
								   stride=stride, 
								   padding=padding)
self.interp = e2nn.R2Upsampling(self.in_type, 
								scale_factor, 
								mode='nearest', 
								align_corners=True)
								   

4. 进行一个卷积的unit

x = feature_map
x = e2cnn.GeometricTensor(x, self.in_type)
conv_out = self.conv_nxn(x)
relu_out = self.relu(conv_out)
norm_out = self.norm(relu_out)
pool_out = self.pool(norm_out)
interp_out = self.interp(pool_out)
# 如果这里需要普通操作，例如＋-x÷和普通卷积之类的，需要调用tensor属性
out = interp_out.tenosr
final_out = out * out

5. 一定要有.tensor属性调用，不然会报错，因为非e2cnn的计算操作只针对tensor或者numpy有效

# 如果没有上一步调用tensor属性，则会报错
print(type(interp_out))
print(type(out))
输出：GeometricTensor
输出：Tensor