Deep Learning-based Methods For Chinese Zero Pronoun Resolution

Zero pronoun resolution is one of the main research topics in the field of natural lan-guage processing.The goal for this task is to recover the ellipsis in sentences and generate complete sentences.Pronoun resolution plays an indispensable role in information extrac-tion.The users for information extraction care about the relationships between entities and events,and these components always lie in different parts of the documents.Therefore,there exist various of expressions for these entities,for example some entities could be ex-pressed by some certain pronouns.In order to extract useful information from sentences,it is necessary to resolve pronouns in these documents.Zero pronoun,as a special phe-nomenon of pronoun,is ubiquitous in pro-dropped languages,such as Chinese.For these natural language expressions that are full of ellipsis(zero pronouns),human beings can easily understanding the actual meaning of them,while it is relatively hard for machines.Therefore,zero pronoun resolution plays a key role in Chinese natural language under-standing.Existing approaches for Chinese zero pronoun resolution regard feature vectors as the inputs and train the machine learning classifier to identify the antecedents for zero pronouns.These approaches highly rely on human efforts and they all overlook the se-mantic information when building their models.In recent years,deep learning methods have achieved success in the field of natural language processing and distributed repre-sentation models become more and more popular Compared with the traditional machine learning methods,distributed representation models can utilize deep neural networks to generate abstractive and high-level semantic representations for the specific task.In this thesis,we employ deep neural networks,propose four different models for the task of Chi-nese zero pronoun resolution,improving the overall performance from different aspects.In specific,our thesis research on the following aspects:1.Recurrent neural network-based model for Chinese zero pronoun resolution.Tra-ditional approaches for this task utilize syntactic and grammatical information to build the resolver and overlook semantic information.This is mainly because that zero pronouns have no descriptive information,for instance,the number and gender information,and therefore makes it hard to calculate the similarity between zero pronoun and candidate antecedent at the semantic level.For example,gender mapping and number mapping fea-tures that have been proven to be essential in the resolution of pronouns are unavailable.To better deal with this problem,in this thesis,we propose a recurrent neural network-based zero pronoun resolution model,which utilize the contexts information of the zero pronoun to map it into distributed representation and thus gain the deep semantic rep-resentation of the zero pronouns.When modeling the candidate antecedents,our model can generate both the global and local information for these noun phrases,and utilize this information to help resolve the zero pronouns.Experimental results on the benchmark dataset OntoNotes 5.0 shows that our recurrent neural network-based model surpasses all the baseline systems.2.Memory network model for Chinese zero pronoun resolution.Because of the lack of descriptive information,it is important to represent zero pronouns at the semantic level.Among all the components,the antecedents of zero pronouns are the ones that contain the necessary information to interpret these gaps.Therefore,to better represent zero pronouns,we propose a memory network model that learns the representation of zero pronouns by utilizing the potential candidate antecedents.With the help of this,our model can finally choose the correct antecedent.Experiments illustrate the effectiveness of applying the memory network.3.Attention-based neural network model for Chinese zero pronoun resolution.Exist-ing approaches for zero pronoun resolution regard all the words equally when representing the zero pronoun.However,some words in the associated sentence play a much more im-portant role in representing zero pronouns than other words.Based on this observation,we propose an attention-based neural network model that learns the importance of each word by utilizing the attention mechanism and then generate the representation vector for zero pronouns by considering the difference of different words.We gain better perfor-mance on the benchmark dataset than all previous baseline systems.Experimental results on the OntoNotes 5.0 shows the efficiency of the attention mechanism,our attention-based neural network model beats all the baseline systems all across the board.4.Deep reinforcement learning model for Chinese zero pronoun resolution.Tradi-tional approaches for this task are all pairwise models.For each time,these model only predict the classification result of the current antecedent-zero pronoun pair alone and thus overlook the decisions of other antecedents.Meanwhile,the local decision cannot influ-ence further classification decisions.To better solve the above problem,we propose a novel deep reinforcement learning model for this task,expending the traditional classifi-cation model into the sequential decision process.For one given zero pronoun,we regard all its candidate antecedents as the sequence and choose all the possible antecedent in this sequence.Experimental results show that the reinforcement learning-based approach can increase the overall performance compared with the traditional pairwise models.In summary,in this thesis,we propose solutions for different challenges of Chinese zero pronoun resolution task,researching on the problems that need to be addressed and increasing the performance of the resolver in the benchmark dataset.More specially,we focus on the following aspects:utilize recurrent neural network to model relations be-tween different candidate antecedents;utilize contexts and potential antecedents to gener-ate representations of zero pronouns;employ attention mechanism to explore importance of different words;and expend the traditional pairwise model into the sequential decision process,identify all the antecedents of a given zero pronoun incrementally.Finally,we hope that our research could provide help for researchers in the area of pronoun resolution and also other fields of natural language processing.