Extracting Chatbot Knowledge From Online Discussion Forums

A chatbot is a conversational agent that interacts with users in a certain domain or on a certain topic with natural language sentences. Normally, a chatbot works by a user asking a question or making a comment, with the chatbot answering the question, or making a comment, or initiating a new topic. Most existing chatbots consist of dialog management modules to control the conversation process and chatbot knowledge bases to response to user input. Typical implementation of chatbot knowledge bases contains a set of templates that match user inputs and generate responses. Templates currently used in chatbots, however, are hand-coded. Therefore, the construction of chatbot knowledge bases is time consuming, and difficult to adapt to new domains.This paper presents a novel approach for extracting high-quality pairs as chat knowledge from online discussion forums so as to efficiently support the construction of a chatbot for a certain domain. Given a forum, the high-quality pairs are extracted using a cascaded framework. First, the replies logically relevant to the thread title of the root message are extracted with an SVM classifier from all the replies, based on correlations such as structure and content. Then, the extracted pairs are ranked with a ranking SVM based on their content qualities. Finally, the Top-N pairs are selected as chatbot knowledge. Results from experiments conducted within a movie forum show the proposed approach is effective.Contribution of this study can be summarized as follows: 1. Perhaps for the first time, this paper proposes using online discussion forums to extract chatbot knowledge. 2. A cascaded framework is designed to extract the high-quality pairs as chabot knowledge from forums. It can optimally use different features in different passes, making the extracted chatbot knowledge of higher quality. 3. Experimental results show that structural features are the most effective features in identifying RR and author features are the most effective features in identifying high-quality RR.