Content assessment in intelligent computer-aided language learning: Meaning error diagnosis for English as a second language

Language practice that includes meaningful interaction is a critical component of many current language teaching theories. At the same time, existing research on intelligent computer-aided language learning (ICALL) systems has focused primarily on providing practice with grammatical forms. For most ICALL systems, content assessment of a learner response is limited to comparing it with a target response through string matching. But this severely restricts exercises that can be offered because expected responses must be tightly controlled. Yet the task-based activities that language instructors typically use in real language-learning settings require both form and content assessment in answer evaluation. Thus, there is a real need for ICALL systems that provide accurate content assessment beyond string matching.; This thesis addresses that need by taking an empirically driven approach to the exploration of content assessment. This work argues that while some meaningful activities are too unrestricted for ICALL systems to provide effective content assessment, where the line should be drawn on a spectrum of language exercises is an open question. At one extreme of the spectrum, there are tightly restricted exercises requiring minimal analysis in order to assess content. At the other extreme are unrestricted exercises requiring extensive form and content analysis to assess content. While full natural language understanding is beyond the scope of current technology, this thesis explores activities in the space between spectrum extremes.; The primary source of material for this exploration into ICALL content assessment is a corpus of language learner data collected for that purpose. The corpus is comprised exclusively of responses to short-answer reading comprehension questions by intermediate English language learners. Responses to these questions are ideal for developing and testing an approach to content error diagnosis because they exhibit linguistic variation on lexical, morphological, syntactic and semantic levels, but they have definable target responses that capture what it means to be correct.; The corpus is one of the first known to be annotated with diagnoses of meaning errors. Diagnoses were developed from analyzing the learner data and adopting an annotation scheme based on target modification. This corpus provided invaluable insight into the considerations necessary for developing an approach to diagnosing meaning errors.; Because variation is possible across learner responses in activities in the middle ground of the spectrum, any degree of content assessment must be flexible and support the comparison of target and learner responses on several levels including token, chunk and relation. This thesis presents an architecture for a content assessment module (CAM) which provides this flexibility using multiple surface-based matching strategies and existing language processing tools. This thesis shows that content assessment for middle ground language activities is feasible using shallow NLP strategies. Detection of meaning errors approaches 90% in the test set. It also shows that diagnosis of meaning errors is feasible using an approach that relies on machine learning, though additional testing with a larger corpus is needed. By developing and testing this model, as well as exploring the middle ground of activities, this work begins to bridge the gap between what is practical and feasible from a processing perspective and what is desirable from the perspective of current theories of language instruction.