Four major downstream tasks of BERT
1. Sentence pair classification task
MNLI: A large-scale classification task with the goal of predicting whether the second sentence contains, contradicts, or is neutral relative to the first sentence.
QQP: A binary classification task that predicts whether Quora is equivalent on two semantic questions.
STS-B: Semantic Text Similarity Benchmark, a collection of sentence pairs extracted from news headlines or other sources. Then it is marked by score, indicating how similar the two sentences are semantically.
MRPC: Automatically extract sentence-pair compositions from online news sources, with human annotations to indicate whether sentences in a sentence-pair are semantically equivalent.
SWAG: Adversarially Generated Context Dataset contains 113k complete examples of sentence pairs for evaluating solid commonsense reasoning. Given a sentence, the task is to choose the most plausible continuation among four choices. Among them, when fine-tuning on the SWAG dataset, the training data is constructed in series according to each input sequence containing a given sentence and possible continuation words
2. Single sentence classification task
SST-2: The Stanford Sentiment Treebank is a single-sentence binary classification task consisting of sentences extracted from movie reviews and human annotations with their sentiments.
CoLA: The Linguistic Acceptability Corpus is also a single-sentence binary classification task with the goal of predicting whether an English sentence is linguistically "acceptable"
3. Question Answering Task
SQuAD: The Stanford Question Answering Dataset is a collection of 100,000 question/answer pairs. Given a question and a passage in Wikipedia that contains the answer, the task is to predict the answer text range (start, end) in the passage.
All BERT fine-tuning methods add a simple classification layer to the pre-trained model and jointly fine-tune all parameters on downstream tasks. However, not all tasks can be easily represented by the BERT architecture, thus requiring the addition of task-specific model architectures.
4. Named entity recognition task
The single-sentence tagging task is also called the named entity recognition task, or NER for short. The common NER datasets include CoNLL-2003 and so on. This task refers to identifying entities with specific meanings in the text, mainly including names of people, places, institutions, proper nouns, etc., as well as texts such as time, quantity, currency, and proportional values.
Similarly, Bert cannot be fine-tuned by adding a simple classification layer on the NER task, so we need to add a specific architecture to complete the NER task.