Research On Gesture Recognition And Hand Pose Estimation In Augmented Reality

In recent years,with the rapid development of augmented reality(AR)technology,a variety of AR head-mounted displays emerge in the market.This is accompanied by a profound change in the way of human-computer interaction.Traditional interactive methods such as mouse and control handle can no longer meet people’s demand for intelligent interaction.Researchers begin to explore a new human-computer interaction method centered on human.Gesture interaction has become the preferred interaction mode for augmented reality due to its natural and flexible characteristics.Gesture recognition and gesture estimation have attracted wide attention due to their great academic significance and practical application value.This thesis focuses on gesture recognition and hand pose estimation for the actual interaction requirements of augmented reality scenarios.The research mainly includes hardware design,data set creation,algorithm design and model application.Details are as follows:For the task of gesture recognition,a data glove is designed,which collects hand movement data based on Inertial Measurement Unit(IMU)and transmits the data via Bluetooth.The glove can work in any area,which compensates for the camera’s narrow viewing Angle.Based on the data glove,this thesis creates a gesture dataset,and proposes Gesture Recognition Network(GR-Net).The algorithm realizes gesture recognition by learning spatial and temporal features of dynamic gesture data through convolutional neural network and gated recurrent unit.Finally,the effectiveness of GRNET algorithm is verified by experiments,and the algorithm is applied to the virtual UAV control scene to enhance interactive immersion.For the task of hand pose estimation,this thesis collects first-person hand pose images based on AR head-display RGB camera and creates a dataset of hand pose.Then,the Hand Pose Estimation(HPE-Net)algorithm is proposed.The algorithm uses stacked hourglass network to learn the features of hand key points,and further learns the connection relation between key points through graph convolution neural network to estimate the position of hand key points.This algorithm provides the basic theoretical support for gesture interaction.Finally,the superiority of HPE-NET algorithm is verified by comparative experiments,and it is applied to virtual object movement scene to increase the interactive flexibility.