Gesture Recognition And Applications Of Human-machine Interaction In Augmented Reality Environment

Benefiting from the huge development of technologies such as machine learning,artificial intelligence,and pattern recognition,the traditional machine is becoming more multifunctional and having more interaction with human beings.Also,the pattern of human-machine interaction is extending to the human-machine-integration pattern from the traditional human-prior-andmachine-assist pattern.Augmented Reality(AR)is a new research area extended by the development of virtual reality and has a wide prospect in the applications of medicine,machinery manufacturing,military,education and commerce.In AR environment,people can interact with virtual objects in 3D real scenes but not being confined to 2D images.During the interaction in AR environment,people and machine support each other to achieve the goals.This interaction mode greatly promotes the development of the next generation of human-machine interaction.In the AR environment,a free and natural interaction is necessary in the human-machine interaction.Because the gesture has the better clarity and larger amount of information in communication than the mode of appearance which is more direct and natural with less limitation,the gesture interaction becomes one of the most common interaction modes in human-machine interaction.Gesture interaction includes a variety of complex gestures,such as static complex gestures,dynamic complex gestures and complex gesture tasks.During the human-machine interaction,the machine not only needs to precisely recognize the single static or dynamic gesture from the user and give the control command according to the recognition result,but also needs to recognize and understand a series of complex gesture tasks expressed by the user.Only in this way,the user can be ensured to have a natural human-machine interaction experience.This dissertation towards the gesture interaction and applications of human-machine interaction,proposes the recognition algorithms for the static and dynamic gestures,and the recognition and understanding algorithms for multiple complex dynamic gesture tasks and vague dynamic gesture tasks.The algorithms solve the problems of lacking in training samples,low accuracy in gesture segmentation,difficulties in recognizing complex and vague gesture,and high computation complexity in dynamic gesture recognition.The algorithms are applied in the Augmented Reality-assisted industrial assembly training and orthodontics wire-bending training system.The major works and innovations of this dissertation can be summarized as follows:(1)An extraction method for the static gesture appearance features and a human-machine interaction control algorithm based on decision trees.The images of hands are segmented by the depth data and the fingertips are located by k-curve algorithm.The concentric circles are used to extract the appearance features and the decision trees are applied to classify and recognize the static gestures.This algorithm can fast recognize the gestures and ensure the invariant of rotation and scale.Also,the skeleton key point data captured by Kinect are employed to recognize the dynamic control gestures.(2)A novel dynamic gesture recognition algorithm of directional pulse coupled neuron network(DPCNN)is proposed for real-time interactions of human-machine integration.The gesture recognition is converted into the shortest path problem by transforming the feature matrix to an undirected graph.The DPCNN can select the firing direction by giving different excitations to neighbor neurons and reduce the effects of useless neurons.(3)An algorithm for multiple complex dynamic gesture recognition and understanding is proposed.We consider that the complex dynamic gesture task can be decomposed to a series of hand operations and each operation can be decomposed to several continuous actions.Each action is related with a typical gesture so that the complex dynamic gesture task can be identified and predicted by recognizing the sequences of gestures.During the gesture recognition,we iteratively optimize gesture boundaries by the score probability density distribution to reduce interferences of invalid gestures.(4)An algorithm for vague dynamic gesture recognition and understanding is proposed.For the vague dynamic gesture recognition,a Temporal Logical Relation(TLR)network is developed to sparsely sample individual frames and learns the temporal logical relation.Also,a new type of sparse optical flow called Focus Grid-Optical Flow(FGOF)is proposed to reduce the computational cost and time.(5)The Augmented Reality-assisted dynamic gesture interaction systems for human-machine integration are designed.We design the Augmented Reality-assisted industrial assembly training and orthodontics wire-bending training system for multiple complex dynamic gesture tasks and vague dynamic gesture tasks respectively to validate the reliability of proposed dynamic gesture recognition algorithms.