Study notes 32-pytorch code realizes global covariance pooling GCP

To realize the torch covariance matrix, please refer to the blog

GCP concept
The covariance matrix describes the correlation between a group of random variables. The method of global covariance pooling is to select the value that can represent the distribution of feature map data by calculating the covariance matrix (second-order information) of the feature map.
The torch.cov() function can realize the calculation of the covariance matrix by itself, and the specific calculation process is officially explained by the figure below.

According to the calculation formula of covariance, simply write code to implement GCP, and apply it to the classification layer of CNN network.

class CovariancePooling(nn.Module):
    def __init__(self, input_c: int, squeeze_factor: int = 4):  # 输入通道数，超参压缩因子（默认为4）
        super(CovariancePooling, self).__init__()
    def forward(self, x):
        scale = torch.zeros(x.shape[0],x.shape[1],1,1)
        x=x.cpu()
        for i, batch in enumerate(x):     #用于将一个可遍历的数据对象 (如列表、元组或字符串)组合为一个索引序列
            for j, ch in enumerate(batch):
                ch = ch.view(-1)#重新定义矩阵的形状,自动调整这个维度上的元素个数，以保证元素的总数不变。
                vecs_np = ch.detach().numpy()
                cov = np.cov(vecs_np.T)
                cov = torch.tensor(cov)
                scale[i, j] = cov
        return scale.cuda()

Solution It
is worth noting that vecs_np = ch.detach().numpy()
since the PyTorch Tensor variable of the type to be converted has a gradient, if .detach()it is directly converted to numpy data without adding it, the calculation graph will be destroyed, so numpy refuses to perform data conversion. This is actually a reminder to developers. If you do not need to preserve gradient information when converting data, you can add .detach()calls before variable conversion.