An investigation of the impact of automated software testing tools on reflective thinking and student performance in introductory computer science programming assignments

This research examined the benefits of automated software testing on student performance and levels of reflection. Edwards (2004) theorized that the increased grade performance that he observed in students who used his Web Center for Automated Testing (Web-CAT) software was the result of increased levels reflective thought in students, such as the reflection-in-action described by Schon (1983). The participants in this study consisted of 144 students in introductory Java programming courses at the University of West Florida. Students were invited to use the Web-CAT software-testing tool for three software projects in the middle of a semester. Students were not required to write their own test cases. Instead, the testing tool used researcher-supplied test cases to evaluate student code and provide immediate feedback to the students. At the end of the semester, student self-reported levels of reflection were measured using Kember et al.'s (2000) reflective thinking survey. Students who used the software were grouped into three usage levels: none, low and high. The only significant difference in the levels of reflective thought among any of the usage categories was a lower level of reported understanding for the high Web-CAT usage level. Average student project performance also increased significantly for those in the high usage level. Students, instructors, and administrators could benefit from the adoption of such automated testing software and may see improvements in student performance even without student-written test cases. More research is needed, however, to determine if student-written test cases would provide an increase in student reflective thought.