Research And Development Of Virtual Musical Instruments Based On Multimodal Human-Computer Interaction

With the development of hardware and software technology about computer, human-computer interaction(HCI) has been constantly updated. The traditional human-computer interaction is still using the graphical interface based on keyboard and mouse.These interactive devices require the user to master skills of a certain extent, and limits the user to stay at the computer side. It's not direct, natural and convenient enough to human's interaction. And with some new interactive equipment and technology coming out, multi-modal human-computer interaction(MMHCI) become the hot topic in the field of human-computer interaction.Multi-modal human interaction is being used in almost every areas of people's learning and life. Using multi-modal interaction and virtual reality technology, we can develop new interactive application that allows us to do natural human-computer interaction activities more easily. The main research of this paper is to study the integration of somatosensory technology, computer vision technology and other interactive technologies, to design and develop a new human-computer interactive model integrating somatosensory-based interactions, gestures and traditional keyboard and mouse interaction. Based on this model,I develop a virtual traditional music instruments playing system. The main research contents of the following aspects:1) Study and discuss multi-modal human-computer interaction in the major conceptual issues, multi-modal human-computer interaction of information fusion technology, and the latest multi-modal human-computer interaction technology,like somatosensory,2) Study of Kinect somatosensory interactive technology:research the Kinect somatosensory technology principles, techniques, etc. And somatosensory-based interactive technology application.3) Object recognition and tracking based on image recognition:using EmguCV image processing library, track the position of hand. Using such means to identification and to do human-computer interaction.4) Kinect-based, research of the complementary combination of image recognition-based interaction and somatosensory interaction, design and develop a virtual bells playing system using this interactive mode.5) human-computer interaction based on Leap Motion's hand accurately identify, research Leap Motion mode and hand recognition technology, design and development of virtual musical instruments system based on hand movements which can be more natural, and has richer interactive features.