Distilling Bert-based Model For Natural Language Understanding

In recent years,with the development and application of deep learning in the natural language processing(NLP),people enjoy more and more diversified intelligent language services.In particular,smart assistants served on various terminal equipment have started a new way for people to interact with machines.However,the existed smart assistants on the market are pre-defined and trained by the software developers,and users cannot migrate them to the desired domain directly.Aiming at such industry pain points,a universal chatbot platform will be helpful and valuable with customized and personalized services.Users can use this platform to create exclusive dialogue chatbots that meet their own business needs without understanding any basic knowledge and methods for implementation.For the COVID-19 epidemic which is still not under effective control,the public health service based on this platform can provide convenient health services for users who are quarantined at home,and collect public health information to help the government control the spread of the epidemic.Natural language understanding(NLU),an important part of dialog system,consists of two sub-tasks: intent classification and named entity recognition,which extract and identify the intention and entity information in user's input respectively.NLU provides the system with the ability to understand human language,hence its accuracy and speed are very important to the user's experience when using the system.Pretrained language model BERT(Bidirectional Encoder Representations from Transformers)is trained on a largescale unsupervised corpus and has made a breakthrough in many NLP tasks.In the research of NLU,the Joint BERT based on BERT achieves a very good performance.However,a BERT-based model is often computationally expensive,which makes it difficult to work efficiently in a real-time task like dialog system.Aiming at the above requirements and problems,this thesis designs and implements the NLU module and its interfaces in the platform by means of deep learning methods,and applies it to the scenes of public health service dialogue system.Firstly,this thesis realizes the Joint BERT model as the original model and uses Tiny BERT,a knowledge distillation method for BERT model,to transfer it to a lightweight model as the objective model for NLU.When the user's dataset is small,a data augmentation method based on word replacement is introduced to improve the distillation performance.Secondly,this thesis makes a systematic analysis of the public health service and designs the intent labels and entity labels according to the requirements scenario.Finally,this thesis conducted experiments on public datasets in the open domain and on medical datasets for COVID-19.The results show that the NLU module proposed in this thesis can predict accurately and efficiently whether as a general module in the platform or a special module in the public health service.