Dialogue Management In Cognitive Conversational Systems

Dialog manager is the core module of task-oriented dialog systems.It has two important missions:dialog state tracking(DST)and dialog policy optimization.The role of dialog state tracking is to estimate the user's goal,and the dialog policy determines how to reply to the user.Partially observ-able Markov decision process(POMDP)provides a good theoretical foun-dation for data-driven methods of dialogue management.A variety of dia-logue management methods based on machine learning and deep learning have been proposed.However,in practical applications,these methods are also facing two important challenges:training data sparseness and do-main extension and transfer.The main reasons for data sparseness are as follows:the difficulty in collecting task-oriented dialog data,the complexity of data annotation,the variety of dialogue,and the sparse feedback signals of on-line dialog policy learning.Domain extension and domain transfer are also called the extension of the cognitive boundary in dialogue systems.In the first part of this thesis,to address the two challenges of data sparseness and domain extension,two novel dialogue state tracking methods are proposed based on the ideas of bridging knowledge-driven and data-driven methods and structured deep learning.To address the problem of data sparseness,this thesis proposes a hy-brid dialog state tracking method based on the combination of rule-based methods and statistical methods:constrained Markov Bayesian polynomial(CMBP).In the CMBP framework,the DST model is defined as a set of polynomial functions that satisfy certain constraints in which human prior knowledge and domain knowledge are encoded.Under reasonable assump-tions,the optimization problem of DST model can be transformed into an integer linear programming problem.By solving the integer linear program-ming problem,we can obtain a set of DST models that conform to the prior knowledge and domain knowledge,and then we can use the labeled data to select the best model.The experimental evaluation on DSTC-2/3 data sets shows that the proposed method has a significant performance improvement compared with traditional rule-based models and statistical models in the case of extremely sparse training data or relatively sufficient training data.To address the problem of domain extension,this thesis proposes a uni-versal end-to-end dialog state tracking model.The key to solving the prob-lem of domain extension is to make the model has the ability to adapt to the dynamic extension of domain ontology and can transfer knowledge between slots and domains.In this thesis,a universal dialog state tracking model based on structured deep learning is proposed,which can automatically con-struct a relational graph according to the domain ontology and model the relationship between slots with graph neural networks.When the domain ontology changes,only the relation graph changes,and the model can still run on the new relation graph.The model not only supports the dynamic extension of domain ontology in the model structure but also achieves the best performance on some data sets of dialog state tracking.In the second part of this thesis,to address the two challenges of data sparseness and domain transfer,two novel dialog policy optimization meth-ods are proposed based on the ideas of bridging knowledge-driven and data-driven methods and structured deep learning.To address the problem of data sparseness,especially sparse feedback signals of on-line dialog policy learning,this thesis proposes an online pol-icy optimization framework,i.e.companion learning.Traditional rule-based policies are usually reliable in the predefined scope,but they are not adaptive.While the policies based on reinforcement learning can be automatically op-timized according to user's feedback,their initial performance is often poor and the learning is not efficient.The companion learning framework pre-sented in this thesis combines the two methods.The rule-based policy acts as a "teacher" and guilds the data-driven reinforcement learning policy by providing example actions and extra rewards.Experimental results show that this method can significantly improve the initial performance and learning efficiency of on-line policy optimization compared with other methods.To address the problem of domain transfer,this thesis proposes a uni-versal policy optimization model AgentGraph based on structured deep rein-forcement learning.It consists of some sub-networks,each one correspond-ing to a node of a directed graph,which is defined according to the domain ontology including slots and their relations.Each node can be considered as a sub-agent.The same types of nodes share parameters.In order to model the interaction between agents,each agent can communicate with its neigh-bors when making a decision.Moreover,when a new domain appears,the shared parameters in the original domain can be used to initialize the param-eters of AgentGraph in the new domain.The results of evaluation on the PyDial benchmark show that AgentGraph not only achieves the best perfor-mance on most tasks but also can be effectively transferred from the source task to a target task.In conclusion,to address the problem of data sparseness,this thesis proposes a series of hybrid models based on the idea of bridging knowledge-driven and data-driven methods,which significantly reduce the dependence of models on large-scale training data.To address the problem of domain extension and transfer,this thesis proposes a series of universal models based on the idea of structured deep learning,which can effectively support slot extension and knowledge transfer.