Semi-supervised learning (SSL)
Semi-supervised learning (SSL). Let the learner automatically use unlabeled samples to improve learning performance without relying on external interaction. That is, under the guidance of a small number of sample labels, it can make full use of a large number of unlabeled samples to improve learning performance, avoiding data resources. At the same time, it solves the problem that the generalization ability of the supervised learning method is not strong when there are few labeled samples, and the unsupervised learning method is not accurate when there is a lack of sample label guidance.
For details, see related reviews and blogs:
[Summary] Semi-supervised semantic segmentation_m0_61899108's blog-CSDN Blog
Long article summarizes semi-supervised learning (Semi-Supervised Learning) - Know (zhihu.com)
Semi-supervised learning - Zhihu (zhihu.com)
basic assumption
- The Smoothness Assumption smoothing assumption: the one who is close to vermilion is red and the one who is close to ink is black, and the two samples have similar characteristics in high-density space, so the labels should be the same. Optimization schemes such as Mixup, Consistency Regularization and Adversarial Learning.
- The Cluster Assumption clustering assumption: In a high-dimensional feature space, samples of the same cluster should have the same label. This strong assumption is actually a special case of Smoothness.
- Low-density Separation Hypothesis: The classification boundary should be in the low-density area of the sample space. This assumption is more of a necessary condition for the above assumptions. If the decision boundary is in a high-density area, the integrity of the cluster and smooth edges cannot be guaranteed. Optimization schemes such as MinEntropy.
Recently, there are two core methods for semi-supervised image classification tasks: Consistency Regularization and Pseudo -Label .
This article briefly introduces the Match series methods in the semi-supervised algorithm: MixMatch (NIPS 2019), ReMixMatch (ICLR 2020), FixMatch (NIPS 2020), FeatMatch (ECCV 2020), FlexMatch (NlPS 2021).
Code: https://github.com/TorchSSL/TorchSSL
MixMatch: A Holistic Approach to Semi-Supervised Learning, NeurIPS 2019
Interpretation: Super semi-supervised learning MixMatch - Zhihu (zhihu.com)
MixMatch paper reading - Zhihu (zhihu.com)
Paper: https://arxiv.org/abs/1905.02249
MixMatch, an algorithm that guesses low-entropy labels for unlabeled examples produced after data augmentation, and mixes labeled and unlabeled data using MixUp. Many semi-supervised learning methods make the model have better generalization ability by adding a loss term to the unlabeled data. Loss items usually include the following three types: 1. entropy minimization, which encourages the model to output high confidence prediction results on unlabeled data; Output the same probability distribution, 3. Generic regularization (generic regularization), encourage better generalization and reduce overfitting. MixMatch achieves good results by fusing existing methods into one loss.
ReMixMatch: Semi-Supervised Learning with Distribution Alignment and Augmentation Anchoring, ICLR 2020
Interpretation: ReMixMatch paper reading - Zhihu (zhihu.com)
【ICLR2020】ReMixMatch - Zhihu (zhihu.com)
Paper: https://arxiv.org/abs/1911.09785
Code: GitHub - google-research/remixmatch
The "MixMatch" algorithm is improved in two ways: distribution alignment and augmentation anchoring. Distribution alignment encourages the marginal distribution of predictions on unlabeled data to approach that of the true labels. Augmentation anchoring feeds multiple strongly augmented versions of the input data into the model and forces each output to approximate the prediction of the weakly augmented version of the same input.
FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence, NeurIPS 2020
Interpretation: Paper notes --- "FixMatch" - Zhihu (zhihu.com)
FixMatch code analysis - Zhihu (zhihu.com)
Paper: https://arxiv.org/abs/2001.07685
FixMatch is simple and effective. The main innovation lies in the combination of consistency regularization and pseudo-labels, and the prediction of weakly enhanced unlabeled images to generate pseudo-labels, which is used as a supervisory signal for strongly enhanced unlabeled images.
FeatMatch: Feature-Based Augmentation for Semi-Supervised Learning, ECCV2020
Interpretation: [Semi-Supervised Learning] FeatMatch: Feature-Based Augmentation for Semi-Supervised Learning_Mahou Shochu Blog-CSDN Blog
Paper: https://arxiv.org/abs/2007.08505
Proposed in FeatMatch: Learning to refine and enhance input image features through soft-attention of representative prototypes extracted from features of other images.
The comparison between traditional image-based data augmentation and feature-based data augmentation is as follows:
Traditional image-based augmentation suffers from the following two limitations:
- Operating in image space limits possible transformations to texture or geometry within the image.
- Operating within a single instance, it is not possible to use knowledge of other instances, whether inside or outside of the same class.
Some algorithms using Mixup only partially solve the second limitation, because mixup only operates between two instances, such as ICT MixMatch, ReMixMatch. On the other hand, Manifold Mixup approaches the first limitation by performing Mixup in feature space , but only for simple convex combinations of two samples.
In order to solve these two limitations at the same time, this paper proposes a new method FeatMatch, which can refine and enhance image features in the abstract feature space instead of the image space. In order to effectively use the knowledge of other classes, by performing clustering in the feature space to condense the information of each class into a set of prototypes. Then the image features are refined and enhanced by the information propagated from the prototypes of all classes.
Prototype Selection: K-Means performs prototype extraction at each epoch, and the feature refinement and enhancement module updates existing prototypes with newly extracted prototypes in the training cycle.
Feature Augmentation: After selecting a new prototype set, refine and enhance the features of the prototype set through soft-attention.
FlexMatch: Boosting Semi-Supervised Learning with Curriculum Pseudo Labeling, NeurIPS 2021
Interpretation: FlexMatch paper reading - Zhihu (zhihu.com)
代码：GitHub - TorchSSL/TorchSSL: A PyTorch-based library for semi-supervised learning (NeurIPS'21)
FixMatch uses a predefined constant threshold for all classes to select unlabeled data useful for training, without considering different learning states and difficulties in judging different classes . To address this issue, this paper proposes Curriculum Pseudo Labeling (CPL), a curriculum learning method that exploits unlabeled data according to the learning state of the model. The core of CPL is to flexibly adjust the threshold for different classes at each time step, allowing informative unlabeled data and its pseudo-labels to pass through . CPL introduces no extra parameters or computations (forward or backward propagation). Apply CPL to FixMatch, and name the improved algorithm FlexMatch. The threshold will gradually increase with the training of the network, and different categories have different thresholds.
