Continuous Chinese Sign Language Recognition With Kinect

Sign Language is the main method of communication among the deaf and mute. A communication gap forms between healthy community and deaf group, because most healthy people can't understand sign language. Sign language recognition can promote the communication between the deaf and the healthy. It's significant to improve the life of the deaf. The development of computer vision and machine learning technique provides a new approach to sign language recognition. By analyzing the video sequence, we can extract proper features to describe sign gesture, build a proper model for each sign, and translate the video into text at last.Kinect for windows which developed by Microsoft is used as input device. The depth image and skeleton estimation provided by Kinect are input information. First, hand tracking algorithm with depth images and color images is studied. A hand track method is proposed by combining the depth information and color information. Second, various feature descriptors for sign language are studied. A new feature descriptor is proposed for hand shape description. Third, SVM(Support Vector Machine), ELM(Extreme Learning Machine), HMM(Hidden Markov Model) classifiers are used for sign language recognition with the above features. Fourth, a Kinect-based Chinese Sign Language Dataset which contains color data, depth data and the skeleton position is published; Fifth, the latent dynamic conditional field and weighted feature back propagation neural network are used to label the transition frame. Finally, we finished the small vocabulary continuous Chinese sign language recognition.The main content of this thesis is as follows:First, we introduce the background and significance of the study of Chinese sign language recognition, describe the difficulties of the study, and introduce the structure of the thesis.Second, we study data acquisition with Kinect and image preprocessing. We use the Microsoft Kinect V2 SDK to get the depth and the color video data, do some image preprocessing to track the hand precisely, get the hand region. We also get the skeleton position data as the raw input.Third, we study the feature extraction and machine learning algorithm used in the sign language system. We extract the Hu and HOG as the shape feature. In order to decrease the computational complexity and increase the accuracy, A new simple and efficient shape feature:APF(Area Proportion Feature) is proposed for hand shape description. A Chinese sign language Kinect dataset is established and published. Different features and different classifiers are compared on the sign language task with this dataset.Fourth, we study the sequence segmentation algorithm which is suitable for continuous sign language recognition. CRF(Conditional Random Field) and LDCRF(Latent-Dynamic Conditional Random Field) are studied, BP neural network-based sequence segmentation is proposed. The performances of the two algorithms are compared. We have done some experiments on continuous sign language recognition with several sentences.At last, we sum up our work on sign language recognition and expect the future work on sign language recognition.