Automatic Evaluation Of Answer Quality Based On Distributed Representation

The answers robot of the college entrance examination is an application of artificial intelligent question answering technology in the actual scene,in ad dition the automatic evaluation of answer quality is also an important question related to it.This paper studies the automatic evaluation of answer quality for the Short Answer Questions(SAQ)in the history entrance examination.Early automatic evaluatio n systems of SAQ rely on structured data,using some rules to match the student answers and references.Later,the researchers based on the string,vocabulary and shallow semantic features to calculate and match the student answers and references.However,the rule-based approach has bad universal property,and the shallow semantic features cannot express the true semantics of the text accurately.The distributed representation based on deep learning maps the text to the semantic space,which has been successfully applied to a variety of natural language processing tasks in recent years.This paper studies the method of automatic evaluation of students' SAQ answers based on distributed representation and compares them with the traditional evaluation methods based on feature engineering.The contents of this paper are as follows:Firstly,the traditional machine learning method has been used to integrate the various features to calculate the similarity of the student answers and the references as the basis for the evaluation,and provide a strong baseline for the automatic evaluation method of answer quality based on the distributed representation.We analysis the co-occurrence features of words and characters,the correlation features of information retrieval model and the semantic features of deep learning,and use Ranking SVM to fuse these three types of features to calculate similarity and select features.As the lengths of reference answers and student answers are inconsistent,extending references can effectively improve the evaluation performance of model.Secondly,we have fused matching calculation based on distributed representation and pair-wise supervised ranking learning model to a framework,and according to the matching degree of student answers and references learned a ranking model.We use the existing text vector representation method to train the paragraph distributed representation of references and student answers,use the cosine similarity,the similarity matrix model,and the tensor model to calculate the matching degree of student answers and references,and then evaluate the student's answer according to matching degree.Due to the limitation of experimental data,this paper constructs the dummy data for training.Experiments show that the performance of our model is comparable to the features engineering method.Thirdly,we have constructed a quality estimation model based on deep neural network on small-scale data.We use a bidirectional LSTM and a CNN-LSTM to model the student answers and questions to obtain their distributed representation vectors respectively,and then calculate the cosine similarity of the two vectors for the answer quality estimation.We train and test the model on the existing real small-scale data,and adjust the size of training data to explore the experimental data size on the impact of model training.