Representation Learning Based Information Extraction

Information Extraction is a significant task in natural language processing,aiming at extracting entities,relations and events from text and forming the structured outputs.A fundamental problem in information extraction is how to effectively represent the truth semantic meaning of texts.The importance of learning semantic meaning of texts could be reflected from two aspects.First,teaching a machine learner to detect the knowledge of text requires a deep understanding of the semantics of texts.Semantic representation of texts could be viewed as the bridge to communicate between computer and human being.Second,machine learning approaches have achieved promising performances on a vari-ety of information extraction subtasks.As the performance of a machine learner heavily depends on the choice of data representation.Traditional methods mainly use the discrete representation as input.And then employ some machine learning methods to obtain task related results.These approaches rely heavily on the quality of the feature selection and the earlier natural language processing tools outputs.In recent years,distributed feature representations have been extensively used in different tasks in the field of artificial intel-ligence.Compared with symbolic representations,distributed representations not relies on any task-specific resources,and can be much more naturally combined with deep neu-ral network models for learning high-level task-specific semantic representations through layer-wise representation abstraction.In this thesis,we focus on designing different dis-tributed representation models based on the properties of each subtask in information ex-traction to improve their performances(Attention Model,Deep Memory Network,Long Short Term Memory Network,Convolutional Neural Network,Tensor Network).It is well accepted that information extraction contains different subtasks,On this basis,we summarize our research works into the following four aspects in this thesis.1.Multi-Level Cross-Lingual Attentive Neural Architecture for Multi-language Named Entity Recognition.Previous studies only consider the sematic clues in current language.And in fact,different languages usually contain complementary cues about entities.For instance,a Chinese sentence“???????_·???”,the word“?”is common in Chinese but rarely appears as a translated foreign name.However,an English translation of“?”is(“Ben”),which provides a strong semantic clue that this is a personal name in the English space.If the English semantic of“?”can be used for Chinese named entity recognition,it will effectively improve the performance.To address this problem,we pro-pose a multi-level attention model,which can map high resource language(English)into low resource language to enrich the latter semantic representation.2.Effective Deep Memory Networks for Distant Supervised Relation Extraction.Previous studies have proposed a semi-supervised framework named distant supervised re-lation,which can overcome the problem of time-consuming in supervised learning.How-ever,the previous methods don't have the capability of explicitly capturing the importance of different context words,but only employ a convolutional neural network to model sen-tence representation as the entity pair representation.In this thesis,we introduce a deep memory to learn the important weights of context words and use them to obtain the entity pair representation.After that,we utilize the dependence of different relations to calcu-late the entity representation towards a relation.The new representation can be regard as features to classify.3.Event Extraction with Sequential and Local Feature Representation.For multi-language event extraction,we discover that many languages lack natural language process-ing tools,such as parser.This shortcoming leads to the traditional feature engineering methods that are difficult to use directly in many languages and achieve good results By analysis and comparison,we find that the sequence and local structure are two language-independent information,and these two kinds of information is critical to identify and classify the event trigger.Based on this observation,we use a Bi-directional long short-term memory to capture both the preceding and following context information of each word.In addition,we employ convolutional neural network to learn salient features in a flat structure.In event extraction experiments,incorporating the hidden representation of Bi-LSTM and CNN as features can significantly improve the performance of event extraction.4.Neural Tensor based Entity Disambiguation.In order to combine the previously learned information and the current knowledge bases,we provide a neural tensor based disambiguation method.In traditional entity disambiguation task,we use different mod-els to learn character representation,semantic representation and context representation towards the given entity.And then through low-rank tensor model to learn the combina-tion of these three kinds of information.After that,we calculate the similarity between the new representation and the candidates,which are some entity pages in wikipedia.We also use the low-rank tensor model to learn the representation of candidates.In the end,we select the correct entity page based on the similarity value.In addition,we formulate an entity pair disambiguation task,which come from knowledge base translation.We learn continuous triple representation with a gated neural network,which integrate entity and relational level information,where the representation of the relationship is calculated by a tensor model.Finally,through the distance function to find the correct translation of the source entity to the candidates.Our experiments show the proposed tensor model can achieve better results compare to previous works in the above two tasks.In summary,in this thesis,we aim at the different challenges of different tasks,and then learn semantic representations of named entity recognition,relation extraction,event extraction and entity disambiguation to improve the performance of corresponding tasks.We hope that our research could be helpful to the researcher in the area of information extraction.