| The pre-trained language models are the cornerstone of natural language processing tasks,which can achieve a high level of language understanding ability through the pretrained task and realize the transformation from text to representation vector.Most of the existing pre-trained language models follow the paradigm of pre-trained and fine-tuning.In the pre-training process,these models undergo self-supervised training on large-scale text corpora to capture the contextual semantics of words.In the Fine-tuning process,they realize model transfer and make the models achieve high performance in different downstream tasks.Traditional pre-trained methods are limited to plain text information,making it difficult to uncover the meaning of specialized words or specific entities.Therefore,researchers further enhance the pre-trained models by incorporating external knowledge(such as a knowledge graph)to improve the performance of the language models.External knowledge can provide more intensive,structured information to the model.Most existing studies are based on external knowledge embedding,joint learning of entity triplets,and entity interpretation to achieve external knowledge integration.They mainly focus on the enhancement of knowledge information in the open domain(such as Baidu Encyclopedia and Wikipedia).However,the data characteristics of the knowledge graph in the closed domain(such as finance,medical,etc.)are significantly different from those in the open domain in the number of nodes and edges,entity coverage,sub-graph density,and other indicators.In this thesis,we analyze that the closed-domain knowledge graph has the characteristics of global sparsity and local density.Most of the classical pretrained language model methods based on knowledge enhancement lack in-depth modeling of knowledge graphs in the closed domain.If directly transfer to the closed domain,the global sparsity will lead to the degradation of model accuracy and make it difficult to improve the application performance of the model in the closed domain.Therefore,based on the data characteristics of the closed domain,this thesis designs a knowledge-enhanced pre-trained language model suitable for the closed domain by combining the density characteristics of the knowledge graph and the multi-level and fine-grained characteristics of entity categories in the closed domain,to make up for the semantic defects of external knowledge.In addition,the practical application scenario has a high requirement on the reasoning speed of the model,so it is necessary to compress the model and reduce the calculation cost of the model.Knowledge distillation is a general method of model compression,but most of the existing schemes are based on BERT architecture for knowledge distillation,ignoring the distillation method of external knowledge.Therefore,we will distill the model from the perspectives of the prediction layer,intermediate feature layer,external knowledge,and so on,and build a distillation framework suitable for our model.To sum up,the main contributions of this thesis include:1.Build a generic pre-trained language model framework for closed domains.Due to the difference in external knowledge characteristics between the open domain and the closed domain,it is not effective to directly transfer the pre-trained language model based on knowledge injection in the open domain to the closed domain.In this thesis,we propose a generic knowledge-enhanced pre-trained language model framework for the closed domain by leveraging the statistical characteristics of the domain,including global sparsity,the local density of the closed domain knowledge graph,and fine-grained entity categorization.Additionally,we conduct extensive experiments on medical and financial data to demonstrate the efficacy of our framework.2.Propose knowledge enhancement method based on closed domain knowledge graph.In the closed domain,the limited entity coverage of the text corpus and low number of nodes in the knowledge graph impede the knowledge injection ability of the model.To solve the problem of global sparsity of external knowledge in the closed domain,we leverage the local density of the closed-domain knowledge graph to explore the relationship information between entities.The external knowledge is converted into the same knowledge form as the text by the way of path splicing,enabling knowledge and text representations to share the encoder and facilitating more harmonious external knowledge injection.Furthermore,we construct positive and negative samples of varying difficulty using the path distance of the knowledge graph for contrastive learning,thereby enhancing the Knowledge discrimination ability of the knowledge model.3.Propose a knowledge enhancement method based on multi-level entity class information.Knowledge graphs are highly specialized,and entity categories have fine-grained,tree-like hierarchies in a closed domain.In this thesis,we introduce hyperbolic space representation to express the hierarchical structure of entity category information based on Poincaré ball.In this model,entity category and entity representation space are fused by conditional layer normalization,and the knowledge injection ability of entity information is enhanced by the task of hybrid mask prediction and entity category prediction.Experiments show that the knowledge enhancement method based on multi-level entity category information can strengthen the representation ability of entities,make the model have richer semantic information with strong differentiation,and improve the performance of the model4.Design knowledge distillation framework suitable for knowledge-injected pretrained model.In this thesis,we design a distillation framework adapted to the closed domain knowledge enhanced pre-trained language model.The distillation is carried out at the prediction layer,intermediate feature layer,and external knowledge layer.In the prediction layer,we introduce the decoupled dummy tags to obtain more precise label characterization.In the middle feature layer,we realize the adaptive cross-layer distillation from the teacher model to the student model.Furthermore,the proposed distillation framework incorporates a knowledge-oriented distillation method that leverages the relational structure to enhance the distillation effect of external knowledge and avoid the loss of knowledge information.The closed domain-oriented knowledge-enhanced pre-trained language model and its distillation method proposed in this thesis have achieved good results on financial,medical,and other data sets,and are significantly better than the baselines.In addition,this thesis analyzed the contribution and function principle of each module in the model through various experiments such as ablation experiment,visualization,comparative analysis,and case study,and proved the effectiveness of the framework. |
