Improving Sentence Simplification Models Based On Sequence To Sequence Model

Recently,sentence simplification has become one of the most important text generation tasks in the field of Natural Language Processing(NLP).Sentence simplification has many application scenarios in practice.On one hand,it can help people with low-literacy skills acquire information effectively.On the other hand,it also can improve the performance of other NLP tasks,like machine translation,text summarization,intelligent conversation,etc.At present,the main research works of sentence simplification models are based on Sequence-to-Sequence(Seq2Seq)models.However,there are still some problems with the Seq2 seq model currently applied to sentence simplification:(1)Inputs of the model are lack of contextual information.Currently,inputs of the Seq2 Seq model based sentence simplification model are relatively independent word vectors without considering the contextual relationship of words in sentences.(2)The encoder structure of the Seq2 seq model is too simple.The current sentence simplification models are based on the Seq2 seq model of a single encoding stage.However,this model not only cannot effectively extract the feature representation of the input text,but also does not fit the habit of people reading multiple times when reading complex sentences.(3)The decoder cannot effectively use all the information of the encoder.For example,a decoder using an attention mechanism is only connected to the final output of the encoder.(4)The model lacks the ability to analyze the hierarchical structure of sentences.Currently,the ability to analyze sentence hierarchy information is missing in the Seq2 seq model applied to sentence simplification.However,the hierarchical structure information of the input sentence is of great significance for sentence simplification.To solve the problem(1)-problem(3),this article proposes a multi-stage encoder Seq2 seq model(MULTI-STAGE model).In the MULTI-STAGE model,the encoding stage is mainly divided into three stages: the N-gram reading stage,the glance-over stage,and the final encoding stage.In the N-gram reading stage,a convolution operation is performed on the word embedding vector of the input sentence to obtain a convolutional word vector matrix with context.The glance-over stage is based on the N-gram stage,and "glance-over" the sentence to obtain the local coding information and global information of the sentence.The final encoding stage is based on the first two stages,and finally encodes the input text.Meanwhile,this paper uses a weak attention connection method which can help the decoder to make full use of the information of the multi-stage encoder.To solve the problem(4),this article constructs the ON-MULTI-STAGE model.Based on the MULTI-STAGE model,the ON-MULTI-STAGE model introduces an Ordered Neurons(ON)network structure to optimize the glance-over stage of the model.The ordered neurons network can provide sentence-level structure information for the encoding stage of the model.Experimental results show that compared with traditional models and other related sentence simplification models based on Seq2 Seq models,the two improved models MULTI-STAGE and ON-MULTI-STAGE proposed in this paper have better sentence simplification effects.And compared with MULTI-STAGE model,the ON-MULTI-STAGE model is even better with an improvement rate of 4.24% with resepect to the MULTI-STAGE model and 6.46% with respect to the best benchmark Seq2 Seq models in the simplification effects.