Unsupervised comparative learning-pretending to be supervised SwAV

The previously mentioned MOCO and SimCLR mainly focus on increasing negative cases, which is time-consuming and laborious, and SwAV has come back to basics. .

Brief steps

1. The input data of each batch is $x\in R^{N\ast C\ast H\ast W}$ , after different Aug, get$x_1,x_2$
2. The $x_1,x_2$Enter the network and get the output ${with}_1,{with}_2\in R^{N\ast d}$
3. Known K cluster centers, expressed as $C\in R^{K\ast d}$ , calculate the similarity between the output and the cluster center, and get the similarity matrix$Q\in R^{K\ast N.}$ Ideally, the similarity between the sample and its own cluster center is 1, and the other is 0. In fact, it is similar to one-hot label in supervised tasks, but the author found that the effect of soft label is better. In this way, each sample gets a new representation (Codes).
4. Calculating the loss, after having z and q, theoretically z and q produced by different views of the same picture can also be predicted to each other, the author defines a new loss: $$L(z_t,{with}_s)=l(z_t,q_s)+l(z_s,q_t)$$
   其中$$l(z_t,q_s)=-\sum _k{q}_s^{(k)}\log g{p}_t^{\left(k\right)}$$
   $$p_t=\frac{exp(z_t^T{c}_k/T)}{{\sum }_{k^{\prime }}exp(z_t^T{c}_k//\tau )}$$
   So the title says that SwAV seems to be pretending to be supervised. .

At the same time, SwAV also proposed a new data enhancement method to mix views of different resolutions.
Multi-crop strategy includes:
(1) Two standard RandomResizedCrop;
(2) V additional small views.
For example, for the ImageNet data set, in the following code:

1. nmb_crops = [2, 6] means two standard random crops and six small views;
2. size_crops = [224, 96] indicates that the size obtained after the standard RandomResizedCrop is $224\ast 224$ , the size of small views after RandomResizedCrop is$96\ast 96$；
3. min_scale_crops = [0.14, 0.05], max_scale_crops = [1.00, 0.14] means that the scale of small views in RandomResizedCrop is (0.05, 0.14), and the scale in standard RandomResizedCrop is (0.14, 1.00).

color_transform = [get_color_distortion(), RandomGaussianBlur()]
if pil_blur:
    color_transform = [get_color_distortion(), PILRandomGaussianBlur()]
mean = [0.485, 0.456, 0.406]
std = [0.228, 0.224, 0.225]
trans = []
for i in range(len(size_crops)):
    randomresizedcrop = transforms.RandomResizedCrop(
        size_crops[i],
        scale=(min_scale_crops[i], max_scale_crops[i]),
    )
    trans.extend([transforms.Compose([
        randomresizedcrop,
        transforms.RandomHorizontalFlip(p=0.5),
        transforms.Compose(color_transform),
        transforms.ToTensor(),
        transforms.Normalize(mean=mean, std=std)])
    ] * nmb_crops[i])
self.trans = trans

SwAV further narrows the distance between self-supervised learning methods and supervised learning, and the accuracy of supervised learning is only 1.2%. The SwAV here is trained for 800 epochs on a large batch (4096). In the end, the combination of the two methods brought 4.2 points of improvement:

comparing the performance of different self-supervised learning methods when batch_size = 256, SwAV is still the most effective SOTA.

The role of Multi-crop
As shown in the figure below, for self-supervised learning methods, the Multi-crop strategy of 2 160+4 96 is always better than the augmentation of 2*224.