Using immersion and information visualization to analyze human-virtual human interactions

We identify verbal and nonverbal communication as the primary way in which humans try to interact with virtual human interfaces, and then use this result to develop three different approaches for analyzing interactions between humans and virtual humans. Each of these approaches is applied to analyzing interactions with virtual humans for training domain-specific interpersonal skills. In providing new approaches to analyzing virtual humans interfaces, we advance the state-of-the-art in facilitating and training interpersonal interactions with virtual human interfaces.;Verbal and nonverbal communication is identified as the primary way users try to interact with virtual humans by comparing interactions with virtual humans to similar interactions with real humans. In two user studies (n=82), participants elicited the same information from a virtual and real human using verbal communication. However, participant nonverbal behavior indicated participants were less engaged, insincere, and demonstrated a poorer attitude towards the virtual human. These behavioral differences likely stemmed from the participants' difficulty understanding the virtual humans limited expressive behavior.;The Interpersonal Scenario Visualizer (IPSViz) was then developed to enable review, analysis, and evaluation of the communication between a human and a virtual human. IPSViz generates visualizations of a human-virtual human interaction by capturing, logging, and processing the human and virtual human's verbal and nonverbal behavior. A user study (n=27) shows that conducting an interaction with a virtual human and then reviewing that interaction with IPSViz elicits self-reflection on interpersonal skills, including verbal and nonverbal behavior, rapport-building, and communicating clearly under stress.;The next system, the Virtual Social Perspective-taking (VSP) system, enables review, analysis, and evaluation of an interaction with a virtual human from the perspective of the virtual human. The VSP system records a virtual human patients experience of talking to a medical student, and then uses the recording to transport the medical student into the patients body and relive the conversation through her eyes. The student relives the conversation to better understand the virtual human patients perspective and learn to address her and future real patients - fears. The results of a pilot study (n = 16) indicate that VSP encourages reflection on the perspectives of others and elicits self-directed change of behavior in future social interactions.;The last system, IPSVizN, enables review, analysis, and evaluation of trends and outliers in human-virtual human interactions. IPSVizN processes groups of human-virtual human interaction logs to generate summary visualizations of the interactions. An evaluation of IPSVizN with representative end-users found that participants were able to rapidly (within minutes) identify trends and outliers in overall group interpersonal skills, including verbal behavior, organization, completeness, empathy, and communicating under stress. Identifying these trends and outliers without IPSVizN would have required hours of manual effort.