Research On Paraphrase Processing Methods Based On Neural Networks

Diversity(heteronyms)is widely existed in natural language due to we usually use different expressions to convey the same meaning.In natural language processing,we call it paraphrase phenomenon.Paraphrase processing focuses on the recognition and transformation of such "heteromorphic synonymous" expressions in language,which are called “paraphrase identification” and “paraphrase generation” respectively.In text summarization,paraphrase identification can help identify synonymous expressions to refine the summarization;in question answering system,paraphrase generation can help expand query to improve the scope of answer retrieval.Previous studies have shown that paraphrase processing can significantly improve model ability to deal with the diversity of language expression in information retrieval,machine translation,text summarization,question answering system and other tasks.In recent years,with the development and indepth research of natural language processing,paraphrase processing as an important topic of natural language processing has attracted more and more attention.So far,great progress has been made in paraphrase identification and generation based on deep neural networks.In paraphrase identification,the task is to judge whether two sentences are semantically equivalent,and most of existing methods use neural networks to learn sentence representation in a completely isolated manner.Specially,the sentence representation learning adopts sequential methods.However,paraphrase identification aims at identifying the semantic relationship of two sentences,and therefore it is worth exploring how the semantic relevance between sentences affects paraphrase identification and how to model it.Meanwhile,sequential modeling cannot effectively capture the structure information of sentences.In paraphrase generation,the task is to create different expressions that share the same meaning for a given sentence.Existing model training method has limitations in diverse generation,in which the syntactic variations of paraphrased sentences are relatively small.However,in practical applications,there is an increasingly urgent need for diverse paraphrase expressions.Some tasks even require specific syntactic structures,such as style transfer.Therefore,syntactically controllable paraphrase generation has become a hot topic in current research.How to build a paraphrase generation model with diversity learning objectives and introduce syntactic structure as controllable elements to participate in paraphrase generation has became a new challenge.On the other hand,excellent paraphrase generation models rely on a large scale of paraphrase parallel corpus,and their performance will be seriously degraded when facing with new languages or domains.Therefore,paraphrase generation under low resources is an unavoidable problem in the application of paraphrase processing.Faced with the existing problems and new challenges in paraphrase identification and paraphrase generation,this work focus on sentence semantic representation learning in paraphrase identification,training method of diverse paraphrase generation model,design of syntactically controllable paraphrase generation model and solutions of paraphrase generation under low resources.It can be summarized as follows.In paraphrase identification,the key is the learning of semantic representation of two sentences.We consider modeling the relatedness between two sentences,which will help to determine the semantic relationship between them.Therefore,we propose a deep bidirectional interaction learning model for sentence semantic representation.By making each word in a sentence read the information of another sentence,we can obtain the sentence representation that captures the related semantic information between sentences.Furthermore,considering that the semantics of a sentence is determined by its syntactic structure,we propose a sentence semantic composition method based on syntactic structure,and introduce phrase-level interaction learning.Since phrase structure can provide more context information and play the role of disambiguation,the introduction of phrases can not only enrich sentence representation,but also help to establish phrase level association between two sentences,so as to better capture semantic relationship between two sentences.On this basis,we build a neural paraphrase identification model based on syntactic structure,which improves the accuracy of paraphrase identification.In paraphrase generation,the key is to generate diverse paraphrase expressions.The traditional paraphrase generation model uses a single cross-entropy loss function to train a model,which penalizes the generation mismatched with the reference sentence,but also inhibits the generation of diverse paraphrases.In this work,we consider that the reward function of diversity generation can be added in model training to encourage the model to explore diverse paraphrases.At the same time,in order to avoid the semantic deviation caused by diverse generation,we further increase the penalty function of semantic inconsistency.Since the objective functions of diversity learning and semantic inconsistency are discrete and cannot directly participate in network optimization,we use reinforcement learning to combine these objective functions with neural network parameter training.On this basis,we build a paraphrase generation model based on multiobjective reinforcement learning.Compared with baseline systems,the proposed method can significantly improve the diversity of generated paraphrases.Furthermore,this work studies controllable paraphrase generation method that introduces syntactic information to control the structure of the generated sentences,by which the model can generate various paraphrases.A typical method is to extract the semantic representation and syntactic representation from the given input sentence and syntactic exemplar respectively,and then generate the paraphrase sentence according to the styntactic structure specified in the syntactic exemplar.In controllable paraphrase generation,the core problem is the representation learning of syntax and semantics.To solve this problem,we propose a method based on separable representation learning to model the semantics of the input sentence and the syntax of the syntaxtic exemplar.However,the full separation of sentence semantics and syntax is very difficult.Further,we introduce the semantic reconstruction task of input sentence to enhance its semantic representation learning,and the part-of-speech prediction task of syntactic exemplar to enhance its syntactic representation learning.At the decoder,we design gating mechanism to fuse both the syntactic representation with semantic representation,so that the model training can learn syntax.For model evaluation,we construct an evaluation data set including multiple syntactic transformations and evaluate its quality.Finally,experiments show that the proposed method can effectively improve the performance of syntactically controllable paraphrase generation model.Existing controllable paraphrase generation models rely on paraphrase parallel corpora,and threrefore their performance drops sharply when facing with new languages or domains due to the lack of annotated paraphrase parallel corpora.For this problem,this work studies the solution of controllable paraphrase generation under low resources.There are a large number of bilingual parallel corpora at present,which are semantically equivalent and the different expressions of one language corresponding to the other can be regarded as paraphrases.In this work,we propose a controllable paraphrase generation model based on bilingual parallel corpora.We use bilingual data to learn cross-lingual shared semantic and syntactic encoders and paraphrase generation decoder,based on which we build a controllable paraphrase generation model.By doing so,we can transfer semantic and syntactic knowledge learned from bilingual data to paraphrase generation.Experiments and analyses show that the proposed model using bilingual parallel corpus as training data can generate syntactically controllable paraphrases.When a small-scale paraphrase corpus is available,the performance of the bilingual training model can be further improved by fine-tuning.The ultimate goal of natural language processing is semantic understanding.Paraphrase,as a task directly related to semantics,is one of the future challenges of natural language processing.This work focuses on the research and exploration of specific problems in paraphrase identification and paraphrase generation,and proposes corresponding models and verifies their effectiveness.In the future,we will further study the semantic and syntactic representation learning methods in controllable paraphrase generation,and we also study how to use external resources,such as paraphrase phrases and templates,to improve the diversity and controllability of parpahse generation model.