Research On Question Similarity In Question Answering System

In the big data age,it is very important to avoid the interference of the noise data and to obtain accurate and effective knowledge efficiently.Intelligent question answering system is one way for people to obtain information and knowledge.The intelligent question answering system accepts user' query in natural language form,through the internal analysis,retrieval,computation,extraction and other operations,returns the corresponding answer to user in the same natural language form.Unlike the traditional information search tools,search engine for example,the intelligent question answering system returns the only accurate answer,rather than returning a series of related candidate answers,so that the efficiency and satisfaction for user to access knowledge is greatly improved.Intelligent question answering is an interdisciplinary research involving multidisciplinary fields,including information retrieval,natural language processing,machine learning,data mining,artificial intelligence and so on,which has important academic research significance and industrial application value.The intelligent question answering system mainly includes two kinds of architectures,the structure based on free text and the structure based on question-answer pairs.Both of these two architectures are all very dependent on information retrieval technology.Similar question retrieval is one of the research hotspots.The question retrieval technology is based on the analysis of user's query or input question,firstly to retrieve the similar questions whose meaning is close to the user's real query intention from the question-answer document database,and then obtain candidate answers.In order to understand user intent more closely and accurately,semantic similarity calculation plays a key role.Text semantic similarity calculation can be divided into several levels: concept level,lexical level,phrase level,sentence level,paragraph level and document level.This paper mainly discusses the question text semantic similarity in sentence level.Traditional question similarity calculation method is mainly some probability statistical methods,such as vector space model,probability model,translation model etc.,however,the semantic relation between questions is ignored.In this paper,we propose two kinds of models for the semantic similarity calculation,one is based on the traditional natural language processing method and the other is based on the deep learning method.Through experiments on the real data set,these two methods are more balanced and stable with MAP index basically maintaining an average level,and especially deep learning method improves the recall and F1-measure significantly.