Research On Telemetry And Interaction Technology Based On RGBD Camera In Virtual Environment

The difference between the human-computer interaction mode and the traditional twodimensional graphical interface computer interaction mode lies in that the former requires special hardware(such as handle,helmet,etc.)to realize interactions because of its virtual threedimensional environment.To better adapt to such three-dimensional environment,human-computer interaction demands a more natural way(like somatosensory interaction,eye tracking,etc.),especially when users conduct virtual-reality-based psychological researches,from whom irrelevant factors should be prevented as much as possible.Researching natural interaction technologies in virtual environment is of great application value.In this thesis,the RGBD cameras from the Kinect platform is used to collect both color and deep video data stream which are then converted to somatosensory interaction data and telemetry heart rate data.Combined with the WorldViz virtual reality system platform,the somatosensory data is synchronously transmitted to virtual reality applications through wireless LAN information channels.The main work has the following parts:(1)Designed and implemented a somatosensory sensing system based on RGBD camera.The system primarily acquires and outputs two types of data: the somatosensory control data stream and the facial RGB video data stream.The somatosensory control data stream includes the body position,posture,gesture and other data.The facial RGB video data utilizes Kinect's face tracking technology,whose output is used for algorithms like telemetry heart rate.(2)Designed and implemented a non-contact continuous algorithm for heart rate measurement based on color video stream.The color signal of human skin has a slight change in the video stream which indicates physiological parameter information(heart rate,respiration rate,etc.)of the human body.Taking the face region as the ROI analysis region,the signal obtains the heart rate through filtering,channel separation,region division,region evaluation coefficient and blind source signal separation.The introduction of regional division steps and concept of regional evaluation coefficient in algorithm innovation further improves the accuracy.(3)Designed and implemented a wireless LAN interactive information channel system.It connects the somatosensory data sensing system and the WorldViz virtual reality system.Once the Kinect client submits control data or somatosensory data to the interactive information path system according to the designed coding rules,this system transmits the information to the WorldViz virtual reality system based on the network protocol and outputs the result to the virtual reality application after decoding the data.(4)Applied Integrate somatosensory interaction logic into the virtual reality system platform and conducted corresponding tests.For non-contact heart rate measurement part,three factors are investigated: video length,ROI size,and color channel,all of which have an impact on the algorithm.For somatosensory interaction part,in the psychological experiments under three virtual reality environments,the user experience is tested and compared under three different interaction logics which are scene interaction,model interaction and perspective interaction.