[Machine Learning] The difference between supervised, unsupervised, self-supervised, semi-supervised, and weakly supervised

1. Supervised: training with labeled data;
2. Unsupervised: training with unlabeled data;
   K-means, etc.
3. Semi-supervised: Use the model assumptions on the data distribution to build a learner to label unlabeled examples. Usually it is a two-stage training, first use (smaller scale) labeled data to train a Teacher model, and then use this model to predict pseudo labels for (larger scale) unlabeled data, as the training data for the Student model; knowledge
   distillation .
4. Self-supervision: train on unlabeled data, let the model learn the inner representation of the data through some methods, and then connect to downstream tasks, such as adding an mlp as a classifier. However, after receiving the downstream tasks, it is still necessary to finetune specific labeled data, but sometimes you can choose to completely fix the previous layer, and only finetune the parameters of the subsequent network.
   Self-supervised data supervision comes from the data itself. In fact, it is to generate some simple labels by itself, and then learn. In theory, the network can learn some feature information of the image, and then add other tasks on the basis of this model.
   Split the image into regions and scramble it, then let the model restore it to the original image. Much like a jigsaw puzzle.
   Hollow out some areas in the picture, and then let the model make predictions so that the output can restore the input picture as much as possible.
   Rotate the image by a certain angle and let the model predict how many degrees it has rotated. Or rotate the picture into four categories of 0, 90, 180, and 270 degrees, and let the model predict the rotated category.
5. Weak Supervision: Training with labeled data that contains noise.
   The strength of the label refers to the amount of information contained in the label. For example, compared to the segmentation label, the classification label is a weak label . If we know a picture and tell you that there is a pig in the picture, then you need to put Where is the pig, where is the boundary between the pig and the background to find out, then this is a weakly supervised learning problem to learn a strong label if the label is known