Partial convolution && Gated convolution

group meeting discussion thread

---

1. Image Restoration

Image inpainting (inpainting), as the name suggests, is to repair the damaged part of the image. It is an image editing technology that can be applied to removing objects, repairing old photos, image completion (eg, earthquake interpolation) and so on.

2. Partial convolution

Link to the paper: Image Inpainting for Irregular Holes Using Partial Convolutions (2018 ECCV)

The previous deep learning image completion methods used ordinary end-to-end CNN to do it, that is, the damaged image was used as input and the complete image was used as a label for learning. When ordinary convolution ( vanilla convolution ) acts on the damaged area of ​​the image, most of the calculations are wasted, because the pixels in the damaged area are 0 or 1; at the same time, the convolution kernel cannot distinguish between damage and Undamaged regions are not sensitive to the information difference between the two parts.
Pconv participates in the convolution operation by adding a mask mask, which greatly improves the operation efficiency, and distinguishes the pixels in the damaged area from the undamaged area, improving its sensitivity.

Why are traditional convolutions inappropriate for image inpainting [free-form] tasks?
Traditional convolution is to calculate each pixel as an effective value. This feature is used for classification and detection tasks. In these tasks, all pixels in the input image are valid, and then ordinary convolutions extract local features in a sliding window manner. But for the inpainting task, there will be some holes in the repaired image (that is, the pixels inside are invalid), and it is necessary to distinguish the content inside and outside.

Partial convolutional layer:
$$x^{\prime }=\{\begin{array}{ll}{\mathbf{W}}^T(\mathbf{X}\odot \mathbf{M})\frac{\text{sum}(1)}{\text{sum}(\mathbf{M})}+b,& \text{if sum}(\mathbf{M})>0\\ 0,& \text{otherwise}\end{array}$$
where $X$ is the feature value (pixel value) of the current convolution (sliding) window,$M$ is the corresponding binary mask,$W$ is the convolution kernel weight,$b$ is the bias,$\odot$ means pixel by pixel multiplication. For M \mathbf{M}in the first layer of Pconv$M$ , 1 represents undamaged area, 0 represents damaged area.
as the picture shows:

Mask value:
$$m^{\prime }=\{\begin{array}{ll}1,& \text{if sum}(\mathbf{M})>0\\ 0,& \text{otherwise}\end{array}$$
That is, $As\; long\; as\; one\; element\; in\; M$ ​​is 1, then the updated$m^{\text{'}}$ is 1; otherwise, it is 0. as the picture shows:

2.1 Loss

The image with holes is ${\mathbf{I}}_{in}$, the initial binary mask is $M$ , the network predicts${\mathbf{I}}_{out}$, the real image is $I_{gt}$。
Per-pixel losses :
$${\mathcal{L}}_{hole}=\frac{1}{N_{I_{gt}}}\Vert (1-M)\odot ({\mathbf{I}}_{out}-I_{gt}){\Vert }_1$$
$$L_{valid}=\frac{1}{N_{{\mathbf{I}}_{gt}}}\Vert (M\odot ({\mathbf{I}}_{out}-I_{gt}){\Vert }_1$$

$I_{comp}$Represents the original output $I_{out}$But the non-hole pixels are set to the ground truth image.
${Ps}_p^{I^{\ast }}$Represents the image $I^{}$ppof${}^{\ast }$$p$ -layer activation mapping (projection to higher-level feature space using ImageNet pre-trained VGG-16)
Perceptual loss (perceptual loss):

Assuming advanced features ${Ps}_p^{I^{\ast }}$The shape is $H_p{W}_p\times C_p$, can get a $C_p\times C_p$Gram matrix, $K_p$for the ppThe normalization factor of the$p$$\frac{1}{C_p{H}_p{W}_p}$）。

Sytle-loss term（风格损失项）:

Total variation loss（全变差损失）：
$${\mathcal{L}}_{tv}=\sum _{(i,j)\in R,(i,j+1)\in R}\frac{\Vert {\mathbf{I}}_{comp}^{i,j+1}-I_{comp}^{i,j}{\Vert }_1}{N_{{\mathbf{I}}_{comp}}}+\sum _{(i,j)\in R,(i+1,j)\in R}\frac{\Vert I_{comp}^{i+1,j}-I_{comp}^{i,j}{\Vert }_1}{N_{I_{comp}}}$$
总损失：
$${\mathcal{L}}_{total}=L_{valid}+6L_{hole}+0.05{\mathcal{L}}_{perceptual}+120({\mathcal{L}}_{styl{e}_{out}}+L_{styl{e}_{comp}})+0.1{\mathcal{L}}_{tv}$$

2.2 Its application in super-resolution tasks

The input to the network is constructed from low-resolution images by offsetting pixels and inserting holes. The details are as follows:
Let the low-resolution image be $I$ , the size is$H\times W$ , the scale factor is$K.$ _
Let the network input image be$I\text{'}$ , its size is$(K\ast H)\times (K\ast W).$forIIEach pixel in$I$$(x,y)$ , put it in$I^{\prime }$中的$(K\ast x+\lfloor K/2\rfloor ,K\ast y+\lfloor K/2\rfloor )$ , and the correspondingM \mathbf{M}The value at the$M\; position\; is\; 1.$

3. Gated convolution

Paper link: Free-Form Image Inpainting with Gated Convolution (ICCV 2019)

What are the disadvantages of partial convolution?
1. When updating the Mask, it heuristically classifies spatial locations as valid and invalid. No matter how many pixels are covered by the filter range of the previous layer, the mask of the next layer will be set to 1 (for example, 1 valid pixel and 9 valid pixels are treated as the same to update the current mask) , which seems unreasonable.
2. All channels in each layer share the same mask, which limits flexibility. Essentially, partial convolutions can be viewed as non-learnable hard gating of single-channel features.

Partial convolution and gated convolution:

Gated convolution layer：
$$Gatin{g}_{y,x}=\sum \sum W_g\cdot I$$
$$Featur{e}_{y,x}\sum \sum W_f\cdot I$$
$$O_{y,x}=\varphi (Featur{e}_{y,x})\odot \sigma (Gating{}_{y,x})$$
Among them$W_g$ $W_f$Indicates the weight of the corresponding convolution kernel, $I$ is the feature map,$\varphi$ can be any activation function (such as ReLU), and$\sigma$ represents the sigmold function.

Gated convolutions allow the network to learn a dynamic feature selection mechanism for each channel and each spatial location . Interestingly, the visualization of intermediate gating values ​​shows that it not only selects features based on background, mask, sketch, but also considers semantic segmentation of certain channels. Even at deep layers, gated convolutions learn to highlight mask regions and sketch information in different channels to better generate inpainting results.

3.1 Loss

电影器：
$${\mathcal{L}}_G=-{\mathbb{E}}_{z\sim {\mathbb{P}}_z(z)}[D^{sn}(G(z))]$$
风别器：
$${\mathcal{L}}_{D^{sn}}={\mathbb{E}}_{x\sim {\mathbb{P}}_{data}(x)}\left[ReLU\right(1-D_{sn}(x))]+E_{z\sim {\mathbb{P}}_z(z)}[ReLU(1+D^{sn}(G(z)))]$$

3.2 Gconv implementation code

class GatedConv2d(nn.Module):
    """
        Gated Convlution layer with activation (default activation:LeakyReLU)
        Params: same as conv2d
        Input: The feature from last layer "I"
        Output:\phi(f(I))*\sigmoid(g(I))
        """

    def __init__(self, in_channels, out_channels, kernel_size, stride=1, padding=0, dilation=1, groups=1, bias=True,
                 batch_norm=True, activation=torch.nn.LeakyReLU(0.2, inplace=True)):
        super(GatedConv2d, self).__init__()
        self.batch_norm = batch_norm
        self.activation = activation
        self.conv2d = torch.nn.Conv2d(in_channels, out_channels, kernel_size, stride, padding, dilation, groups, bias)
        self.mask_conv2d = torch.nn.Conv2d(in_channels, out_channels, kernel_size, stride, padding, dilation, groups,
                                           bias)
        self.batch_norm2d = torch.nn.BatchNorm2d(out_channels)
        self.sigmoid = torch.nn.Sigmoid()

        for m in self.modules():
            if isinstance(m, nn.Conv2d):
                nn.init.kaiming_normal_(m.weight)

    def gated(self, mask):
        return self.sigmoid(mask)

    def forward(self, input):
        x = self.conv2d(input)
        mask = self.mask_conv2d(input)
        if self.activation is not None:
            x = self.activation(x) * self.gated(mask)
        else:
            x = x * self.gated(mask)
        if self.batch_norm:
            return self.batch_norm2d(x)
        else:
            return x
          

Gated convolution and attention mechanism:
similarities:
1. Contextual information capture: Both aim to capture long-distance dependencies and contextual information of input data by introducing mechanisms to improve the model's ability to understand data.
2. Adaptability: Both the gated convolution and the attention mechanism are adaptive, and can automatically learn weights or importance according to different parts of the input data for weighted processing.
Differences:
1. Application field: Gated convolution is mainly used in image processing tasks in convolutional neural networks (CNN), such as image description generation. The attention mechanism is widely used in models such as recurrent neural network (RNN) and Transformer for processing sequence data, language modeling, machine translation and other tasks.
2. Mechanism: Gated convolution weights the input by introducing gating units (such as update gates and reset gates) to adjust the weight of the convolution operation. The attention mechanism weights different parts of the input according to how relevant the input is to the context by calculating attention weights.
3. Locality: Gated convolution pays more attention to the local area of ​​the input data, and performs weighted processing on the local area through the convolution operation; while the attention mechanism pays more attention to the global context information, and dynamically adjusts the weighting weight according to the relationship between the input and the context.

---

References:
https://zhuanlan.zhihu.com/p/519446359
https://www.cnblogs.com/wenshinlee/p/12591947.html
https://blog.csdn.net/weixin_43135178/article/details/123229497
https://cloud.tencent.com/developer/article/1759006
https://blog.csdn.net/yexiaogu1104/article/details/88293200?ydreferer=aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MzEzNTE3OC9hcnRpY2 xlL2RldGFpbHMvMTIzMjI5NDk3