Signer-Independent Continuous Chinese Sign Language Recognition With Kinect

Sign language recognition has been a hot topic for Human-Computer Interaction.It is mainly used for the hearing-impaired people to communicate with others.Sign language,that is,a language in which hands and arms are used,and head movements,facial expressions,and body postures are assisted in communicating.The task of sign language recognition utilizes the computer vision,pattern recognition,and machine learning to translate sign language into text or voice output.It promotes the normal communication of hearing-impaired people and other social groups.Sign language recognition task can be divided into two categories:isolated and continuous sign language recognition.Isolated sign language recognition is less difficult than continuous sign language recognition.Sign sequence segmentation and sign recognition are two main problems in continuous sign language recognition.Different persons have different hand size and arm length.Their talking habits with signs are not the same.Continuous sign language recognition is still a challenging problem.Furthermore,signer-independent continuous sign language recognition has more research significance and practical value.RGB-D data for sign language is collected with Kinect in this dissertation.The continuous Chinese daily sign language dataset(SDUSign)is established and released publicly.The extraction and fusion of hand shape feature and motion trajectory feature are discussed.Latent Dynamic Conditional Random Fields(LDCRF)is used for sign sequence segmentation,and Hidden Markov Model(HMM)is used for sign recognition.This dissertation focuses on signer-independent continuous Chinese sign language recognition,the main contributions are as follows:First,the research background of the sign language recognition are introducted.The research of sign language recognition is reviewed in recent years.And the existing problems in the study of sign language recognition are analyzed.Second,the Kinect for windows is used as input device to acquire sign language data.Due to the lack of public Chinese sign language datasets.SDUSign dataset is established and published.This dataset involves 3400 sign samples over 40 different isolated signs of 17 signers 5 times and 600 sentence samples over 10 different sentences of 8 signers 10 times.The process of the data acquisition and the information contained in the dataset are introduced briefly.Third,the image preprocessing and the feature extraction are researched.The hand region can be obtained by combining depth images with color images.And the hand shape features and motion trajectory features of signs are extracted.Hand shape features include Histogram of Oriented Gradient(HOG)and Area Proportion Feature.And motion trajectory features include spherical coordinate feature and hand location feature.It is found that the feature description method that combines hand shape and motion trajectory features can significantly improve the sign language recognition accuracy through comparative experiments.Fourth,the algorithm of signer-independent continuous sign language recognition is analyzed.LDCRF is used to segment sign sentence,and sign length constraint is used to optimize the segmented results.HMM is utilized to recognize the sign segments.A grammar constraint probability model based on continuous sign samples statistics is constructed in order to improve sign recognition accuracy at sentence level.A series of continuous sign language recognition experiments are conducted.Experiments show that the proposed method can effectively recognize sign sentences and eliminate the individual differences of samples.The sentence segmentation accuracy reaches 80.61%,the sign recognition accuracy is 86.25%,and the sentence recognition accuracy is 73.75%.The system shows superior performance.At last,the work of this dissertation is summarized.Then the further research direction of sign language recognition is discussed.