| This study investigated using Bayesian networks for assessment and identification of erroneous procedures, known as “bugs”, in student performance on subtraction problems. While such bugs are known to exist, they do not appear consistently, even in a single student's work in a single session. Partly for this reason, the measurement problem of diagnosing specific bugs has seen little progress since subtraction bugs were first described in the cognitive science literature.; Our investigation here was conducted using data from a test of multicolumn subtraction skills given to N = 641 third-, fourth-, and fifth-grade students (VanLehn, 1981). Four alternative Bayesian network architectures were proposed and evaluated in this study. They are referred to as the (1) Binary-Answer Bug network, (2) BinaryAnswer Bug-Plus-Subskill network, (3) Specific-Answer Bug network, and (4) Specific-Answer Bug-Plus-Subskill network. Only bugs were posited as causes of item performance in the two Bug-only networks, (1) and (3). Both bugs and corresponding subskills were assumed as causes in the Bug-Plus-Subskill networks, (2) and (4). Performance of these two alternative approaches was compared in two simulated testing situations: first, where the observed information about students gathered by the test consists only of binary correct-incorrect item data, and second, where the provided information simulates a multiple-choice test, including specific answers to specific items given by the student.; The proposed networks showed good performance in predicting bug manifestations in individual students. The prediction rates using various proposed cut-points were greater than 85% for all four bugs in all four networks, and most of the prediction rates exceeded 90%. Results show that one can improve bug prediction rates in the Bayesian network models by: (1) employing specific answer information, and (2) using subskill nodes in addition to bug nodes. However, the increased bug prediction rates for adding subskill nodes were minimal. Lambda statistics for the classification by all networks were over .95, meaning that proportional reductions in predictive errors were greater than 95% for each bug. Use of different cut-points did not make a critical difference in terms of prediction performance, although the best prediction rates were achieved using the fixed cut-point of .50, especially for the Binary-Answer networks. |
