Quantifying code complexity and comprehension

What factors impact how difficult code is to understand? Small, simple programs are often easier to grasp, but can still be locally ambiguous, visually confusing, or violate implicit expectation. The cognitive complexity of a program is something more than its line length, visual layout, and problem domain. It is a gestalt phenomena that arises when a programmer must perform a specific task on a given program or segment of code. Modeling these phenomena is a challenging real-world problem at the intersection of computer science and cognitive science.;Programmers have been the subject of traditional psychology experiments for several decades, correlating code features with task performance. The recent inclusion of eye-tracking in program comprehension experiments, however, has given researchers a unique window into programmers' cognitive processes. Advances in the state of the art for quantitative cognitive modeling have opened the door for a unique confluence of technique and technology: quantitative, cognitive models of human program comprehension. We present an experiment in which 162 programmers, from beginner to expert, were asked to predict the printed output of ten short Python programs. Different versions of the same program were randomly selected, exposing groups of participants to minor variations in code spacing, naming, and function. A subset of participants performed the experiment in front of an eye tracker, allowing for a combined analysis of reading and response behavior. The observed errors, timings, and keystrokes formed the basis for two program comprehension models: one modeling correct participants' low-level reading and response behavior, and the other modeling incorrect participants' interpretation errors. These models form the foundation for future research in program comprehension, and join the frontiers of quantitative modeling in cognitive science.