Research And Application Of Deep Convolutional Neural Network Based Egocentric Hand Gesture Interaction

The increasingly uprising trend,Virtual Reality and Augmenting Reality,boost the interest of research on egocentric hand gesture interaction.By apply computer vision and computer graphic technology,hand signals are recognized and turned into command to control devices.Using gestures to communicate is instinctive,which is earlier than language and text.Since gesture is frequently used and easily recognized,gesture interaction turn out to be a new interaction approach than keyboard-mouse or touch screen.With the existence of smart glass and other smart head-mounted devices,new algorithms are required for machines to understand the need of human being.Therefore,egocentric gesture interaction enjoys promising future in daily application,which make the research in this paper meaningful.This paper works in egocentric gesture interaction research based on convolutional neural network,including algorithm and application.The main contribution lays on:1.Researching on latest work of computer vision,deep learning especially object detection and object classification,as well as gesture interaction including hand detection,fingertip detection and so on.Based on these research,the paper makes further progress on tasks such as hand detection.2.Based on convolutional neural network,the paper establishes two datasets for this egocentric task,namely,EgoFinger and EgoGesture.EgoFinger shows data with one single finger and EgoGesture contains multiple gestures in egocentric vision.Two datasets are evaluated from the distribution of color and space considering the rationality and contribution of the datasets for the research domain.3.This paper makes research on hand detection,fingertip detection and gesture recognition using convolutional neural network as feature extractor based on the above two datasets.Hand detector and gesture classifier are based on SSD(Single Shot Multi-Box Detector)framework and the fingertip detector is based on coordinate regression with CNN(Convolutional Neural Network).4.With the output of algorithms,a demo application named Egocentric Air-writing is established.The demo application applies the trajectory formed by detected fingertips as input of text recognizer,and applies gesture class as controlling signal.