| A simulation study of inferential analysis was conducted with ten professional intelligence analysts. Using a process tracing methodology, patterns in vulnerabilities were identified when analysts were asked to analyze something outside their base of expertise, were tasked with a tight deadline, and had a large data set.; First, study participants were vulnerable to missing critical information. All the participants were observed to use relatively primitive search tactics, quickly narrowing in on a set of documents through the addition of keywords to an initial query and then never conducting further searches. All of the participants missed some of the nine documents that were identified as high quality by the investigator. The group of four participants who found and relied upon some of the high quality documents took more time, read more documents, and made fewer inaccurate statements in their verbal briefings than the group of four participants who did not.; Next, three sources of inaccurate statements by the study participants were identified. First, participants sometimes relied upon assumptions that would normally be correct, but did not apply in this situation. Second, participants sometimes repeated information that was inaccurate in a document that they had read. Strategies that were aimed at identifying and eliminating inaccuracies were generally resource-intensive and time-consuming, partly because the baseline electronic environment that was used in the study did not provide support for tracking conflicts in the data set. Third, participants were observed to rely upon information that was considered accurate at one point in time, but then was later overturned in subsequent updates. Locating updates is a very difficult task as they can be on themes that are not reflected in the date/title view of the documents in the browser window.; At the broadest level, some of the study participants could be viewed as having prematurely closed the analysis process. There was a large variety in the rationales that were provided by the study participants for how they determined when to stop the analysis. Determining when to stop is difficult given that it is difficult to know what information has been missed—the judgment is based on the absence of information.; The main contribution from this study is a model of potential vulnerabilities in inferential analysis under challenging conditions. These vulnerabilities are informative because they point to a set of challenging design criteria that human-centered solutions to data overload should meet in order to be useful. These evaluation criteria are interesting, in part, because they are so difficult to address. They are not amenable to simple, straightforward adjustments or feature additions to current tools. Meeting these design criteria will require innovative design concepts. |
