Research On Gesture Interaction Technology Of Augmented Reality Electronic Sand Table

The augmented reality electronic sand table uses the augmented reality technology to realize the dynamic construction of the virtual three-dimensional model on the basis of the physical sand table.It displays the three-dimensional topography and sandbox elements in all directions.and also changes the interaction mode of the sand table,which directly affected the commander's command efficiency.Applying gesture recognition to the sand table can provide commanders with efficiency and natural interactions and improve the operational command efficiency.Therefore,this thesis studies the gesture interaction and fingertip tracking technology.The research content mainly includes fingertip detection method,fingertip 3D motion estimation and tracking,gesture interaction based on multi-sensor fusion.Firstly,a fingertip detection method based on template matching and region growth is designed to meet the requirements of real-time and accuracy of sand table interaction system.According to the fingertip geometric feature and the human skin color characteristic,a region growing method based on template matching is used to segment the hand region in the initial image,and then the fingertip template matching is performed on the region to complete the initial fingertip detection.Since the region growing algorithm takes a long time,just matching a certain neighborhood of the current fingertip position is used to reduce the calculation amount in the subsequent detection process to achieve fast and accurate fingertip detection.Secondly,according to the physiological characteristics of the fingertip movement,the two-dimensional motion information of the fingertip can be mapped onto the three-dimensional spherical surface,and a Kalman filtering method based on the three-dimensional mapping of the fingertip motion is proposed.According to the law of fingertip movement,the composite Kalman motion model of fingertip motion is fitted,and the position of the fingertip is predicted and updated to achieve stable tracking of fingertip motion.Finally,this paper implements augmented reality display and gesture interaction on AR smart glasses Hololens platform.For fewer gestures can be recognized by Hololens,a gesture recognition extension method is proposed.Based on the original gestures,the gesture recognition based on EMG signals is extended.According to the different characteristics of the two input modes,designing a decision-level multi-sensor fusion method to improve the robustness of the gesture interaction of the electronic sand table.The research results of this thesis have been finally applied in the augmented reality electronic sand table scene.