Summary
We propose a novel language-independent method to improve the efficiency of grammatical error correction by dividing the GEC task into two subtasks: Error Span Detection (ESD) and Error Span Correction (ESC) . ESD uses an effective sequence labeling model to identify grammatically incorrect text ranges. Then, ESC uses the seq2seq model to take sentences with incorrect span annotations as input, and outputs only the corrected text of these spans. Experiments show that the performance of our method in both English and Chinese GEC benchmarks is comparable to the conventional seq2seq method, and the reasoning time is less than 50% .
1 Introduction
Due to the increasing number of error correction parallel sentences available in recent years, the sequence-to-sequence (seq2seq) model with encoder-decoder architecture has become a popular solution for GEC, which uses source (original) sentences as input and output targets. (Correction) sentence. Although autoregressive seq2seq models help correct various grammatical errors and perform well, they are not efficient enough for GEC. As pointed out in previous work, the seq2seq model takes most of the decoding steps to copy the grammatically correct text range from the source to the target during the inference process, which is the main efficiency bottleneck . If the time of the copy operation can be saved, the efficiency should be greatly improved.
For this motivation, we propose a simple and novel language-independent method to improve the efficiency of GEC by dividing the task into two subtasks: Error Span Detection (ESD) and Error Span Correction (ESC) , as shown in the figure 1 shown. As shown in Figure 1(a), we use an effective sequence labeling model to identify the text range of grammatical errors in the source sentence. Then, we input sentences with incorrect span annotations into ESC's seq2seq model. In contrast to the conventional seq2seq method of correcting complete sentences, ESC only corrects the span of errors (see Figure 1(b)), thereby greatly reducing the number of decoding steps. The Chinese and English GEC benchmark tests show that our method is equivalent to the seq2seq model based on the latest transformer, and the reasoning time is less than 50%. In addition, our method provides greater flexibility to control the correction, so that we can make precise adjustments in various application scenarios.
2. Related work
Recently, many methods to improve the performance of GEC have been proposed. However, with the exception of those adding synthetic error data and Wikipedia revision logs, most methods will cause increased latency. For example, language models and right-to-left (R2L) scoring not only take time to re-score, but also slow down the cluster size during the inference process to correct the model; multiple rounds of (iterative) decoding require repeated runs of the model; BERT-fuse Added extra calculation for model fusion.
Contrary to extensive research on GEC performance, until recent years, little work has been devoted to improving the efficiency of GEC models. One branch of the work is language-dependent methods such as PIE and GECToR. They predict a series of character-level editing operations, including many manually designed language-specific operations , such as changing the verb form (e.g. VBZ → VBD VBZ → VBDV B Z→V B D ) and prepositions (e.g.in → on in → onin→o n ). However, they are difficult to adapt to other languages. The other branch is language-independent models, such as LaserTagger. They learned the vocabulary of editing operations from the training data, so they can use any language. However, their performance is not as good as seq2seq. Our method combines the advantages of the two branches and has efficient reasoning ability compared with the latest seq2seq method.
3. Error span detection
In order to identify incorrect spans, we use a binary sequence labeling model, where label 0 indicates that the character is in the correct span; label 1 indicates that the grammar of the character is incorrect and needs to be edited, as shown in Figure 1(a). We align the tags across the source sentence and the target sentence in the training data. Through character alignment, we can identify the edited text range, so that the edited text range in the original sentence can be annotated as the wrong range.
4. Error span correction
Using ESD, we can identify the text range of grammatical errors in sentences. If there is no error in the sentence, we will not take further measures; otherwise, we will annotate the incorrect span and use the ESC model to correct it, as shown in Figure 1(b).
To avoid misleading ESC due to ESD span detection errors in the inference process, we randomly select the text span in a manner similar to SpanBERT, instead of training the ESC model based on the labeled error span in the training data only. In this way, the ESC model will see various span annotations and learn how to correct during training, thereby improving its robustness: even if the span detected during the inference process is not very accurate, the ESC model It will not fail easily. By aligning the mark of the source sentence and the target sentence in the GEC training data, we can generate training examples with span annotations and modifications, such as the ESC example in Figure 1(b).
5. Experiment
