Explore Unsupervised Structures In Pre-trained Models For Relation Extration

Relation Extraction(RE)is an important fundamental task in natural language processing,and its goal is to extract all relational triples in a given sentence,which can be used to build intelligent question and answer systems and knowledge graphs.Although end-to-end neural network models have now become the mainstream in RE task because of their efficiency and effectiveness,syntactic structures(e.g.the dependency tree)are still widely used to improve the performance.However,since parsing qualities are not stable on different text domains and a pre-defined grammar may not well fit the target relation schema,the introduction of syntactic structures sometimes fails to improve RE performance consistently.At the same time,pre-trained language models(PLMs)based on unsupervised training have become a powerful representation encoder for natural language processing tasks.In this work,we investigate another kind of external structural features,namely unsupervised structures mined from PLMs,and study its effectiveness and application in RE models.The main work and contributions of this thesis include the following three points:Research1.Firstly,we introduce what is unsupervised structure and how to obtain it from PLMs,mainly based on the related interpretability studies.In order to verify the effectiveness of unsupervised structure in RE under the advantage of easier access,we propose to enhance the representation of relation extraction models by directly encoding unsupervised structures through graph neural networks.The final experimental results show that this method is simple but effective,and the unsupervised structure significantly improves the performance of the baseline,and the effect even exceeds that of the manually defined grammar structure dependency tree.Research2.Secondly,we propose an unsupervised structure application method based on smoothed relationship labels.Based on the unsupervised structure acquisition method,the weight of each edge can be regarded as the probability of the correlation between two inter-words.And that is similar to the idea of the relation table in the well designed table-filling RE model.Therefore,we propose to combine them based on that,to optimize the relation distribution and thus smooth the label.Experimental results demonstrate that the effectiveness of this approach while closely combining the unsupervised structure with the RE task.Research3.Finally,we propose two optimization methods for unsupervised structures in RE.Given the special pre-training settings and scalability of pre-trained language models,this thesis based on unsupervised and supervised training respectively,enhances the PLM which to generate structures.This enables the domain-adaptive optimization and knowledge enhancement of unsupervised structures.The experimental results finally show that these two methods are able to enhance the unsupervised structure simply and effectively by using only easily accessible unsupervised text or entity annotated data,and further improve the performance of unsupervised structures in RE task.