Research On Lightweight Neural Relation Extraction Model

Compared with unstructured information,regular structured information is easier to be used by machines.Therefore,the information extraction technology from unstructured information to structured information is a work worthy of in-depth research.Relationship extraction is a task of extracting structured information.It faces the marked entity pairs and determines the semantic relationship between the two entities according to the characteristics of text sentences.In recent years,the method based on deep neural network has achieved good results in the task of relationship extraction.However,with the continuous improvement of the accuracy of neural network method,the consumption cost of relationship extraction task is also increasing.In order to alleviate this phenomenon,this paper focuses on the lightweight of relationship extraction model.The lightweight relationship extraction method can reduce the size of the extraction model,so that relationship extraction is no longer a task requiring high resources.In the design process,this paper introduces the knowledge distillation technology to improve the extraction accuracy of lightweight relationship extraction model,and tries to design a dynamic loss function with cosine similarity under this framework to alleviate the impact of wrong soft labels generated by a teacher in knowledge distillation.The work in this paper is divided into the following three points:(1)A knowledge filtering mechanism is designed by cosine similarity.In the distillation process,the soft labels generated by a teacher is no longer used directly,but the correctness of the data is judged.This mechanism regards the soft label that can express the meaning of facts as a positive soft label and assigns a higher weight to it;for non-positive soft label,a lower weight is assigned.The soft labels filtered by noise can provide more accurate guidance and alleviate the negative impact of wrong soft labels on the extraction results of lightweight relationship extraction model.(2)A kind of lightweight relation extraction model is constructed by using teacher student knowledge distillation model.The framework has two roles: teacher and student.A teacher outputs knowledge to a student,the student trains under the guidance of knowledge,and constantly absorb knowledge to improve itself.The mechanism of knowledge distillation by the teacher and the student can not only enhance the ability of extraction,but also reduce the amount of time resources.(3)With the help of teacher assistant distillation mode,a lightweight relationship extraction model is designed.Compared with the teacher student knowledge distillation model,the model needs two times of knowledge distillation: the first distillation aims to train a teacher assistant.The second is to transfer knowledge under the guidance of the teacher assistant,who guide the student to train.Through the introduction of assistant,the teacher assistant distillation model avoids the problem that the student model is difficult to imitate the teacher extraction mechanism caused by the large gap between the teacher and the student.By matching the performance of the knowledge training model,teacher assistant has a higher degree of softening of knowledge than the teacher,and can give the student more effective guidance.In view of the above work,this paper selects the commonly used dataset for relation extraction for experimental verification.The extraction effect based on teacher student model and teacher assistant model is verified on Sem Eval-2010 Task 8.The experimental results show that compared with the student model,the lightweight model does not increase the amount of parameters to be trained,and only produces small-scale fluctuations in the running time.And for the lightweight relationship extraction model constructed by using the same student model in these two models,the accuracy after lightweight operation is improved by 0.56% and 1.47% respectively.The results show that the distillation mechanism using positive soft labels can make better use of knowledge.The teacher student distillation framework and teacher assistant distillation framework proposed in this paper are two very effective methods to construct lightweight relationship extraction model,which can improve the extraction accuracy without increasing the model parameters and almost increasing the model consumption time.The lightweight model obtained by two lightweight extraction frameworks has strong competitive advantages in scenarios with limited computing resources and high time requirements.These two frameworks can complete the lightweight transformation at less computational cost,and effectively alleviate the problems of the increasing scale of relationship extraction model and the decreasing extraction efficiency.