Research And Implementation Of Virtual Reality Human-Computer Interaction System Based On Multi-sensor Data

With the continuous development of science and technology,especially the improvement and promotion of 5G technology,the application and development of virtual reality technology has been greatly promoted.The outstanding feature of virtual reality technology lies in its immersive experience.Therefore,the current development direction of virtual reality technology is mainly focused on improving the immersive experience of virtual reality.Human-computer interaction is an important part of virtual reality technology.The reality of immersive experience has an important impact.Traditional human-computer interaction mainly uses keyboards,mice,joysticks,game controllers,etc.This kind of interactive technology will greatly weaken the immersive experience of virtual reality technology,so study a more the natural human-computer interaction system plays an important role in promoting the immersive experience of virtual reality.Based on multiple IMU inertial sensors and EMG myoelectric sensors,this paper collects human acceleration signals and myoelectric signals,recognizes human upper limb movements and gestures in real time,and recognizes natural interactive virtual reality scenes through motions to enhance the immersive experience of virtual reality technology.The virtual reality human-computer interaction system implemented in this paper consists of three main parts,namely virtual reality part,human motion recognition part,and communication subsystem of recognition subsystem and virtual reality environment.The main research contents of the thesis include:(1)System hardware environment construction.The system uses the Oculus Rift CV1 virtual reality glasses as the hardware access part of the virtual reality,and is connected to the development platform through the USB interface and the HDMI interface.The sensor in the motion recognition part uses two wireless IMU inertial sensors,two wireless EMG surface myoelectric sensors,and sends data to the data receiver through ZigBee wireless communication technology.The data receiver transmits data to the development platform through the USB interface.A computer with a core processor of Intel Core 15-7400 and a graphics card of NVIDIA GeForce GTX 1060 is used as a development platform for the system.(2)Research on human motion recognition algorithms.The human motion recognition in this paper includes two parts:upper limb motion recognition and gesture recognition,which are used to interact with the 3D UI interface of the character and virtual reality parts in virtual reality.The recognition algorithm research includes data preprocessing,active segment detection,feature extraction and feature analysis,and classifier selection.Finally,the preprocessing process,active segment detection method and feature needed to realize upper limb motion recognition and gesture recognition are determined respectively.(3)Software design of the motion recognition subsystem.The motion recognition subsistem in the real-time human-computer interaction system of this paper is based on the form of concurrent programming,including sensor data collection and transmission based on multi-process.It is divided into data acquisition process and data transmission process and multi-thread based sensor data reception and motion recognition.It is divided into data receiving thread,data preprocessing and windowing thread,active segment endpoint detection thread,feature extraction and classifier identification thread,and recognition result sending thread.(4)Virtual reality scene construction and environment configuration.The virtual reality scenes in this paper are built using SolidWorks,3Ds Max,Unity 3D and other software to build the virtual reality scenes and character actions.Virtual reality configuration is done using the Oculus driver and the Unity 3D Oculus access interface.(5)The realization of human-computer interaction system.The final implementation of the human-computer interaction system relies on the final motion recognition subsystem and the virtual reality environment communication control.This paper realizes the communication control of the recognition subsystem and the virtual reality environment by customizing the virtual reality interactive control protocol and writing the virtual reality interactive control script,thus realizing the final virtual reality human-computer interaction system.(6)Experimental verification.Based on the implemented virtual reality human-computer interaction system,the interaction effect of the upper limb movement and the gesture action on the virtual reality is tested.Verify the virtual reality human-computer interaction system implemented in this paper to improve the virtual reality immersive experience.