The perception of emotions in multimedia: An empirical test of three models of conformance and contest

Information Systems research usually suggests that information received from multiple modalities results in better communication. Media richness theory, for instance, suggests that one of the reasons face-to-face communication is better than a letter or memo is the greater number of cues and channels used. However, using auditory and visual channels to present information in multimedia does not always provide benefits. When the information from both channels represents common information, the result is a unifying perception but when the information is in conflict, the communication can become confused. This study investigates how emotional information in these two channels is perceived when audio and visual information are in a conformance condition—the emotion information of the two media are consistent in both valence and arousal dimensions—and when audio and visual information are in a contest condition—the emotion information of audio and video are on the opposite sides of both valence and arousal dimensions.; A laboratory experiment was conducted with 162 students where 37 of them were from the institute of music. Participants were randomly assigned to view 3 different sets of audio, video and multimedia combinations in both conformance and contest conditions. Using the method of multiple working hypotheses, the effects of audio-visual combinations are examined in relation to three models of perception—channel dominance, integration, and individual differences. In a conformance condition, the perception of emotions follows the channel dominance model where the perception of emotions from multimedia is similar with the perception of emotions from audio stimuli. In a contest condition, the perception of emotions follows the integration model where perception of emotions from multimedia falls between those of the individual media. Comparisons between participants of different gender, cultural background, and musical expertise reveal no differences in the perception of emotions. Thus, an individual differences model is not supported here.; The results suggest that the perception of emotions is adaptive to different conditions of audio-visual combination. This knowledge is useful as a basis for further investigation of information usage and as a guideline for the design of multimedia information.