Research On Retrieval-based Multi-turn Dialogue With Deep Context Modeling

In recent years,with the rapid development of social media and the Internet,conversational systems have attracted the attention of both academia and industry for their highly promising research and commercial value.Domestic and foreign technology industry companies have released their own products in the field of intelligent dialogue,such as Apple’s Siri,Microsoft’s Ice,Amazon’s Alexa,Baidu’s Xiaodu,Xiaomi’s Xiao Ai,Alibaba’s Xiaomi,and so on.The emergence of these intelligent dialogue products has brought great convenience to the lives of hundreds of millions of people.The goal of dialogue systems is to use computer technology to enable natural language communication between computers and humans.End-to-end chatbots are generally classified into retrieval-based methods and generative-based methods.Retrieval-based approaches generally provide a set of candidate responses,and each time the response is retrieved from the candidate response set.Compared with generative-based methods,retrieval-based counterparts are often superior with fluent and informative response,and easy to evaluate,mainly serving as the core of many real-world chatbots and assistants.However,research on retrieval-based multi-turn dialogues still faces difficulties in dialog context understanding,corpus scarcity,and discourse structure modeling.Therefore,how to further deeply model the contextual information becomes the key to enhance retrieval-based multi-turn dialogues.In this work,we focus on the problem of retrieval-based multi-turn dialogues.We propose corresponding solutions to the challenges of domain adaptation,interactivity,and logic of dialog texts in terms of three aspects: pre-training objectives of language models,modeling of multi-round dialogues interaction,and modeling of discourse structure.Firstly,in order to improve the model’s semantic understanding of dialogues,a domain-adapted pre-training strategy next utterance prediction goal is proposed.Secondly,a model framework for retrieval-based multi-turn dialogue is proposed by combining the advantages of interaction-based methods and pre-trained language models.Experiments and analyses are conducted on three public datasets to verify the effectiveness of the proposed method.And the performance of the model is evaluated on limited data.The experimental results show that the model can still yield a promising performance in low-resource scenarios,which shows the great potential in real-world applications.In addition,the specific discourse structure distinguishes dialogue context from other type of text.In existing approaches,dialogue discourse relations are often ignored,which cause inadequate modeling of the overall semantics of the dialogue context.To better explore the discourse structural features,and improve the logic and reasoning ability of model,this paper propose a Dialogue Discourse-aware Graph Convolutional Model based on discourse relations.We perform discourse parsing on dialogue context and use graph convolutional network to carve out the relations between utterances.Experiments are conducted on the dialogue reasoning dataset Mu Tual,and it is proved that the introduction of discourse relations can effectively improve the reasoning ability of the model.