The Research On VR Piano And Gesture Recognition Based On Leap Motion

With the development of computer,three-dimensional display,human-computer interaction and other technologies,virtual reality technology continues to progress and mature.Especially in recent years,with the popularization of smart phones and the birth of VR BOX,a wave of VR products have formed in the world.Many famous companies have launched their VR products,and VR content has also grown substantially.Due to the creation of realistic virtual scenery and environment,the cost and difficulty are often much lower than the construction of the real world.Therefore,VR technology is rapidly expanding into many fields such as military training,experimental teaching,industrial design,architectural display,medical anatomy,and virtual games.This paper takes the virtual piano as an example to discuss the application of VR technology in music teaching,and focuses on the human-computer interaction technology in VR.The research work and results of this article are summarized below.The paper uses the HTC Vive kit and the Leap Motion sensor fixed on the helmet as the hardware platform.At the same time,a virtual piano is developed using Unity3D and related Stream VR plug-ins and Leap Motion plug-ins as software platforms.The virtual piano uses the Cube component to compose a virtual keyboard,and records the virtual piano keyboard playing event by compiling the Cube component being approached and pressed or releasing the script response function,and further realizes the virtual piano sounding by means of MIDI.In order to enhance the performance of the performance,the paper also implements the keyboard flame of the particle flame system,MIDI document save and replay,and the virtual piano attribute change settings with gesture recognition.In this dissertation,four kinds of hand gesture recognition related to the virtual piano attribute setting are realized by using real-time spatial position information of fingers and palms collected by Leap Motion sensor.The proposed gesture recognition algorithm consists of five modules:data preprocessing,gesture motion extraction,gesture motion framing,gesture meta action encoding,and hidden Markov chain calculations.The data preprocessing module uses debouncing and smoothing filtering to achieve signal denoising.The gesture action extraction module extracts a single gesture from the continuous lengthy data types one by one through a sliding window algorithm,and further passes the meta action frame and encodes as input to the hidden marker.Chain computing module for model training or real-time motion recognition.The immersiveness of the 3D virtual piano developed by the thesis is much higher than that of the 2D virtual piano displayed on the screen of a common computer.It can be played naturally with both hands,and the sense of comfort and fluency is much stronger than the 2D virtual piano played with the mouse and keyboard.The gesture recognition algorithm implemented in this thesis can be used not only for virtual piano attribute setting,but also for other VR products,which has a wide range of popularization and application value.