Research On Chinese Sign Language Recognition With Kinect

The hearing-impaired people communicate with others by sign language and mainly use the hand and arm to express. Simultaneously, the head movements, facial expressions and body language are also used when necessary. Sign language recognition task utilizes the pattern recognition technique to realize the communication between the hearing-impaired people and normal people. In detail, by analyzing the movement features of the hand and arm, the recognition task is finished after the classifier processing the movement features. This article mainly researches the Chinese sign language recognition.Kinect is used for data acquisition in this article, which brings the change in interactive way. Combined with the depth data and the body skeleton joints data, we research the feature extraction and fusion of sign language. As to the isolated word recognition, ELM algorithm is used and its performance is prior to SVM; as to the continuous sign language, the CRF and LDCRF, which are used to process natural language sequence, are tried to solve the segmentation and recognition problem. Our main contributions are as follows:Firstly, the analysis of the research background and significance of the sign language recognition problem have been taken, then we take an overview of the status of the sign language recognition at home and abroad as well as the main problems about the main content and chapters framework of this article.Secondly, the sign language data acquisition and preprocessing based on Kinect are analyzed. The main ways of sign language data acquisition are by camera, data glove or other sensors. Avoiding the constraint of the data glove and other sensors, we utilize the Kinect V1 and V2 with their SDK to realize more natural human-computer interaction. Based on the properties of the sign language expression, we choose the raw data of the hand, thumb, wrist and elbow skeleton joints as the sign language movement information in 3D space.Thirdly, the feature extraction and classification of isolated Chinese sign language are analyzed. The feature extraction is of vital importance for recognition accuracy, which is also an important part of pattern recognition. Based on the depth images and the body skeleton joints provided by Kinect, the 3D movement trajectories features and the hand shape features of the right single hand and the both hand are extracted, then we consider the single feature and the fusion features as the input of our classifier. The SVM and ELM are analyzed for classification, and ELM has a better performance. To verify the effectiveness of the selected features and the classifiers, one dataset including 20 isolated words is established. The experiments results also illustrate that the hybrid features improve the precision apparently, especially for the 8 class of Chinese sign language with the HOG and spherical coordinates feature, and the recognition precision achieve 96.06%.Fourthly, the algorithm of continuous Chinese sign language recognition is analyzed. The popular algorithms for continuous sign language recognition in international and its strength and weakness are briefly introduced. Then, in detail, the principle of CRF and LDCRF is introduced, and we explore the model establishment and the parameter optimization algorithm from three aspects:feature function, parameter estimation and model inference. Four common Chinese sign language sentences are used in experiments part, and the algorithms have realized segmentation of the continuous sign language initially.