Evaluation Framework For Voice Intelligent Agents

Voice Intelligent Agent(VIA)is defined as an intelligent agent that provides services to users through the natural language interaction.This thesis explored an evaluation metric,developed an evaluation guideline,designed and validated an evaluation tool,and finally constructed an evaluation framework for VIAs.Evaluation metrics are measurements of the evaluation quality.In addition to objective performances and subjective feelings,this thesis emphasized the importance of users' cognitive process in VIA evaluations and compared the differences of three think-aloud protocols in VIA evaluations: retrospective,classical,and interactive think-aloud protocols.Evaluation guidelines are important references for the verification of VIA design factors.This thesis proposed a classification guideline for different-aged users.This guideline consisted of three dimensions: content labels,interaction behaviors,and morality intelligence.This guideline was validated under three interaction styles: text,voice and mixed interaction.Evaluation tools can improve cost-effectiveness of evaluations.This thesis designed VIARS as a tool for experts and general users based on the classification guideline.Considering the differences of user expertise and system recommendations,this thesis validated the usability of VIARS in classifications for different item types.Results indicated that retrospective,classical and interactive think-aloud protocols were respectively beneficial for collecting user experience and causal explanation utterances,recommendation and prediction utterances,and problem formulation and recommendation utterances.In addition,both traditional and retrospective think-aloud protocols collected more cognitive process data than interactive think-aloud protocol.Therefore,according to different needs for cognitive process data,evaluators can select corresponding think-aloud protocols.As for the classification guideline,this thesis validated that different levels of all three dimensions obtained different classification results,indicating the validity of this guideline.The mixed interaction group obtained less strict classification results.Therefore,evaluators should conduct the classification work with the same interaction style as the VIA's future application scenario.Then,the VIARS tool was designed based on this classification guideline.It includes five modules: dialog generation,voice-to-text,content analysis,results presentation,and process recording.It also allows users to determine the final classification decisions.Results of the VIARS usability experiment indicated that experts achieved higher consistency in classification results,while general users relied more on VIARS recommendations.VIARS is more useful in evaluating rule-based items,while experts' decisions are required in evaluating experience-based items.In addition,the VIARS tool designed for experts should provide standardized recommendations instead of defensive recommendations.In conclusion,this thesis developed an evaluation framework consisting of evaluation metrics(cognitive process data acquired by think-aloud protocol),evaluation guidelines(a classification guideline)and evaluation tools(VIARS),and served as theoretical references for the practice of VIA evaluations.