Research On Sign Language Recognition Based On Trajectory Movement And Hand Shape Features

Sign language is the main modality of communication in deaf and mute society. It is also one of the most important ways for communication between normal people and the deaf. The deaf can not communicate with the normal smoothly for the sake of the lack of training in sign language for the normal. To address the problem, lots of researchers have devoted lots of research efforts to sign language recognition since the 1990s. The goal is to transform the sign language to the text or speech by computer programming, so that the normal people can understand them. Hence, sign language recognition has a significant social impact.In sign language recognition, there are two critical problems. One is how to collect and design the robust sign features, while the other is how to model the features and recognize it. To address the first problem, researchers adopt the data gloves and color gloves before, but now most of them turn to somatosensory devices like Kinect for collecting skeletons and RGB videos to design the trajectory and hand shape features. To deal with the second issue, lots of machine learning methods are leveraged from in speech recognition, such as DTW, SVM, GMM, HMM, CRF, ANN, and so on. Working on the two problems, this thesis can be summarized as below:Firstly, sign language features include trajectory features and hand shape features. Trajectory features include the direction, the shape, and the position of the trajectory. The direction can be describes by histogram of the oriented displacement, which di-vides the 3D space into three 2D planes to concern the oriented displacement on them. And the combination of the histogram is used as the final feature vector. The shape can be described by shape context, and we should sample the points by density before designing it, which makes the trajectory more smooth. As for position, we utilize the distances between the non-hand joints and the hands. In hand shape description, af-ter preprocessing the video, we obtain the local images which contain the hands, and then we extract the HOG feature in the images. At last the sequence of HOG features describe the sequence of the hand shapes.Secondly, we propose a method called flexible HMM(FHMM) which can auto-matically determine the hidden states of the HMM in terms of the characteristic of the sign language. In order to achieve the adaptation, we divide the hand shape sequence into many sub-sequence by the difference of the successive frames in the sign video. And they will share the similar features in the same sub-sequence. Different features are suitable for the different models due to the different description. Hence, we should conduct late fusion. In this thesis, we propose two kinds of late fusion methods. One is to average the probabilities by frame and add them as the final probability, and the other is to model the output probabilities again. The experiments demonstrate that the FHMM methods can obtain better performance than the classical methods and the better late fusion method leads to the better recognition accuracy.Furthermore, we build a real time sign language recognition demo system and inte-grate the above algorithms to the system. Besides, we also introduce the data collection system and demo system.