Grammatical Error Correction Based On Sequence Generation

In recent years Grammatical Error Correction(GEC),as a Natural Language Processing(NLP)task that aims to edit the original text,has received remarkable attention with wide applications.Its goal is to identify as many errors as possible and successfully correct them,while keeping the original correct words unchanged without bringing in new errors.For the specific GEC models,NMT-based approaches that use sequence-tosequence generation models become the preferred solution for the GEC task.While as sequence-to-sequence models generate corrected results from scratch,they may suffer from generation errors and over-correction problems.Thus some approaches mainly based on sequence tagging models are proposed.They generate correction results by predicting edit operations first and then applying these operations onto the original sentences.Thus the problems in sequence-to-sequence models can be avoided.But they are often dependent on human-designed or automatically generated lexical rules and vocabularies,which limits their generality and transferability between different languages.Moreover,when it comes to corrections which need longer insertions,most of these sequence tagging methods rely on iterative corrections,which can reduce the fluency.On the other hand,by constructing multi-stage GEC systems that connect different GEC models sequentially,the performance on GEC can be improved from another aspect.Among these GEC systems,by adding an extra spelling checker model which edits every tokens in the original sentence,the performance of Chinese GEC models can be improved.Nevertheless,compared to single-stage GEC systems composed of one single GEC model,multi-stage GEC systems introduced a lot more parameters.When it is combined with methods that improve NLP models by incorporating pre-trained language models,the problem of parameter scale will become more serious.Besides,the whole GEC system will be harder to train.In this thesis,from the perspective of improving the sequence generation process,we proposed two methods to solve these two different problems in GEC tasks respectively,action guided sequence generation model and parameter efficient multistage sequence generation:1.From the perspective of model structure,we combine the pros and alleviate the cons of both models by proposing a novel action guided sequence generation model,Sequence-to-Action(S2A)model.It is based on the original sequence-tosequence model with an additional S2A module.With this S2A module,in each prediction step,we jointly takes the source and target sentences as input,and is able to automatically generate a token-level action sequence before predicting each token,where each action is generated from three choices named SKIP,COPY and GENerate.Then the actions are fused with the basic seq2seq framework to provide final predictions.We conduct experiments on the benchmark datasets of both English and Chinese GEC tasks.Our model consistently outperforms the seq2seq baselines,while being able to significantly alleviate the over-correction problem as well as holding better generality and flexibility in the generation results compared to the sequence tagging models.2.From the perspective of optimizing muti-stage GEC systems,we propose a "Single Pre-trained Model,Multiple Adapters"(SPMA)strategy,in which one single large-scale pre-trained language model is shared and fine-tuned for multiple different functions of various parts in a multi-stage Chinese GEC system simultaneously.To implement different functions while controlling the parameter scale,we introduce multiple groups of trainable lightweight adapters.During training,only adapters are optimized while the parameters of the pre-trained Chinese language model is frozen.We conduct sufficient experiments on the benchmark dataset of the Chinese GEC task.And the proposed system outperforms the current stateof-the-art results by a large margin.Together with further analysis,we prove the effectiveness and efficiency of the proposed method.