Improving Pre-Trained Language Representations With External Knowledge For Spoken Language Understanding

Recently,pre-trained language models,represented by BERT(Bidirectional Encoder Representations from Transformers),have demonstrated great performance in spoken language understanding.However,those language models can only establish contextual associations at literature level,and lack of a rich external knowledge to improve their performance.To solve this problem,we mainly carry out the following 2work:(1)We introduced external knowledge composed of Word Net and NELL(NeverEnding Language Learning)for BERT to improve its performance in spoken language understanding.In the process of integrating external knowledge through attention operation,we introduced token-level intent features into the model,to improve the richness of feature information that the model can utilize.At the same time,the introduction of token-level intent features avoids the accumulation of misinformation in the network.Additionally,to enable the model to capture the positional information between different knowledge features in the next attention operations,positional embeddings are introduced for the knowledge features.(2)Since spoken language understanding consists of 2 interrelated subtasks,which are intent detection and slot filling,we take this feature into consideration and design a joint training mechanism that can integrate knowledge features.So that the model can be trained on both subtasks at the same time.This joint training mechanism combines self-attention with stack-propagation.The model will introduce context information for the knowledge features through self-attention operation.The features produced by the model in intent detection will used to improve the model's performance in slot filling.By this way,the model will be able to capture the implicit correlation information between the 2 subtasks.Finally,we conduct multiple sets of experiments on 2 public datasets,ATIS and Snips.Through the experimental results and the specific analysis of 2 cases,we verified that the above work could improve the performance of BERT in spoken language understanding.