Gesture Detection And Recognition In Application Of Human-computer Interaction

Computer revolutions have popularized a more intelligent lifestyle worldwide.Nowadays,the interaction between humans and computers is becoming more often and even an integral part of people's life.And as a main way of realizing this interaction,wearable devices have drawn lots of attention in the society.Body languages,such as gestures,are believed to be the initial and most natural way of communication in human history.Dynamically detecting and recognizing gestures can help facilitate the human-computer interaction,and thus is an important topic in the academia.In recent years,much work has been done on improving the performance of convolutional neural networks(CNNs),which were firstly inspired by the neuron system of beings.Compared to traditional methods,CNNs can automatically extract features from raw images by training on large labelled datasets to recognize patterns in complex problems.Great success have been seen in applications such as scene character detection and recognition,signs and obstacles detection for auto-driving systems,eyes detection and localization.The work in this thesis is focusing on the detection and recognition of first-person view gestures as well as the fingertip detection afterward.Work has also been done on applying these techniques in developing an egocentric view dynamic gesture interaction system.Specifically,the contributions of this thesis include:1.The definition of a set of number based gestures that can be understood easily according to the research on cultures of different countries and the establishment of two labelled datasets(EgoFinger and EgoGesture)that contain first-person view colored gesture videos under different scenes and light conditions.2.The proposal of a CNNs based algorithm that can detect and recognize gestures in the image constantly and accurately.3.The proposal of a fully CNNs and heat map based algorithm that can resist the noise in image backgrounds to significantly improve the speed and accuracy of fingertip localization.4.The development of a Qt framework based interaction system for air mouse controlling and writing.The system has integrated the proposed gesture detection and recognition algorithm and the fingertip detection algorithm.