Research On Model Design And Compression Methods Of Dialogue System Based On Transformer

The intelligent dialog systems,which reserach tasks include dialogue generation,dialogue matching,dialogue state tracking,and dialogue action recognition,is one of the research hotspots in natural language processing.At present,the research on intelligent dialogue systems mainly focuses on the improvement of model performance in various tasks.However,the basic encoders used in different tasks are still based on Recurrent Neural Network(RNN)or Convolutional Neural Networks(CNN).The Transformer model is able to capture the relationship between vocabularies within a sentence and has been proven to have a stronger encoding ability than RNN or CNN in natural semantics.Transformer model is not suitable for encoding long text,while a sample of dialog usually has Multiple dialogue rounds,so the Transformer model cannot be directly applied to dialogue tasks.In addition,the amount of calculation and memory usage of the Transformer model is too large to be widely applied in practice.To solve the above problems,we conduct research on the application of the Transformer model to dialogue tasks.The specific contributions of this article are as follows:1)We propose a model named Mem-Transformer which suitable for encoding dialogue text.The model transfers information between different conversation rounds through the memory network and reduces the amount of calculation through information compression.Finally,it surpasses other models in multiple conversation tasks including dialogue matching,dialogue state tracking,and dialogue action recognition.2)We propose a compression method of the Transformer model.The model uses sliding windows to compress information to reduce information losses and it uses collaborative training of language models to maintain the original sentence information.Finally,experiments show that the compressed model can reduce memory consumption and Calculation amount at the premise of maintaining semantic encoding capabilities.