Research And Implementation On An Augmented Reality Multimodal Interaction

Augmented reality (AR) overlay computer-generated virtual information into the real world, thus enhancing people’s perception and interaction of reality environment. Multimodal interaction provides a more expressive, easier way to understand, more efficient and more robust human-computer interaction (HCI) mode. In augmented reality field, using a multimodal interaction technology can enhance people’s perception, cognition and communication skills. Simultaneously, it can also improve interaction efficiency and user experience. Therefore, multimodal interaction is a very important research issue in augmented reality.This paper deeply analyzes the concept of augmented reality and multimodal interaction, and then conducts a detailed study on the characteristics of augmented reality concept models and multimodal interaction models. Focusing on former modal’s shortcomings and limitations, this paper presents a novel multimodal interaction model under augmented reality from the practical point of view. This novel model uses the virtual perception modal, indirect perception modal and direct perception modal to closely connect users, augmented reality systems, and real-world environment as a whole, which can promote the transfer of information among the three modals. User can interact simultaneously with virtual objects and real world through these three perception modals. This novel model improves the efficiency of human-computer interaction, and provides a more natural and real user experience. Finally, using this model, this paper designs and develops a multimodal educational game system under augmented reality and the use of cars and fireworks as two instances to verify the validity of novel model.