A Hand Gesture Interaction System For EAST Based On Kinect And Fingertip Recognition

As the 3D models of the EAST are very complex and have too many components,for large-screen interaction,the traditional interactive devices such as the mouse and keyboard need to be touched at all times,which are inconvenience to the presenter.At the same time,unlike the common small screen,it makes people feel a sense of strong spatial differences not a sense of spatial consistency when interact with large screen.Moreover it will make people feel tired when interact with it for a long time.Besides,3D model interaction is more complicated than 2D graphic user interaction,for the complex EAST 3D model,the traditional interaction way has limitations,complexity and fatigue.So we design a free,direct interaction system—EAST 3D model hand gesture interaction system for user to interact with 3D model in order to decrease spatial differences,reduce the complexity of the interaction,and avoid long-term fatigue.The research of the system is divided into three parts:requirements analysis,algorithm research and realization,system implementation and testing.During the requirement analysis stage,we introduce the basic information of EAST device briefly,which leads to the interaction requirement of EAST 3D virtual model.Then the structure of EAST model is analyzed and introduced,and the research and analysis of interaction needs are carried out in the form of questionnaires.Through the research and analysis,we designed six type of gestures,such as,move,click,pick,rotate,drag and zoom.At the same time,we propose a dynamic gesture recognition algorithm based on Kinect and fingertip recognition by studying the references of gesture recognition at home and abroad.The implementation of hand gesture recognition algorithm includes four stages:data acquisition,hand region segmentation,feature extraction and gesture recognition.At the data acquisition stage,we first use the Microsoft Kinect(v2)sensor to acquire image frame,then,we extract the roughly hand area of the RGB color image by skeleton data and depth data of the Kinect.Then,remove the background of the hand area by Gaussian Mixture Model background subtraction algorithm,and use the elliptical skin color model algorithm to segment the skin area,after that,smooth the hand area and reduce noise area by the morphological operation,finally we get the accurate hand area.Before the fingertip feature extraction,the contour of the hand area is extracted firstly,and the palm position is calculated by calculating the weighted average of the center of the hand contour and the center of the Huffman Circle.At fingertip recognition stage,we propose a dynamic hand contour sampling distance method according to the hand area scale in image instead of a static sampling distance,then we calculate the K-curvature of the sampling point to get the fingertip candidate point,and finally through the fingertip and palm of the position and distance factors to exclude the non-fingertip point to get the fingertip point.At gesture recognition stage,we use a Finite State Machine method,that is,by using fingertip features and palm features among the continuous image frames as the state machine input,the fingertip and palm position and the number of fingers are main factors to achieve the state transition.Finally,the EAST model gesture interaction system is implemented based on the Unity3D platform by using the gesture recognition algorithm.System functions include model dismantling,subsystem viewing,three views,scene roaming.And the background adaptability analysis and dynamic gesture recognition rate detection are carried out based on this system.The results show that the algorithm has strong background adaptability and high dynamic gesture recognition rate.