User testing of a prototype visualization-based information retrieval system

The VIBE (Visual Information Browsing Environment) prototype system developed at the University of Pittsburgh allows users to select documents from a retrieved set which is graphically represented as geometric icons within one screen display. While the mathematical modelling behind the VIBE system is well known, user interaction with the system is not. This investigation tested the VIBE designer assumption of VIBE being a tool for a smart user and asked two research questions: (1) What are the effects of the different levels of user expertise upon VIBE usability? and (2) What are the effects of user expertise upon usability when VIBE's non-traditional interface is compared with a more traditional text-based interface? The three groups were novices, online searching experts, and VIBE system experts which totaled 31 participants. The traditional retrieval package to which VIBE was compared was askSam. Usability was operationalized by Shneiderman's five measurable human factor goals: (1) system familiarity time, (2) task performance timings, (3) task errors, (4) system feature retention, and (5) subjective satisfaction. Overall, level of expertise had little impact on the performance of typical information retrieval tasks. Significant differences were found for some measures and tasks not only with the novices and experts, but also with the system used for some of the tasks. In particular, VIBE-askSam task error comparisons were statistically significant. System feature retention showed that participants selected appropriate features to complete tasks, but did not solve the tasks correctly. Hence, the task timings improved, but overall effectiveness with the system did not. Subjective satisfaction measures showed that higher percentages of novice and online participants selected the more positive categories for askSam than VIBE. The results suggest that users experienced difficulty in developing a new mental model for understanding VIBE and this factor should not be underestimated when designing an information visualization system. The study concludes with VIBE system interface recommendations and an agenda for future research.