Study On Sign Language Recognition Based On Commercial Wi-Fi Channel State Information

This thesis studies the sign language recognition problem based on Wi-Fi Channel State Information(CSI),implements a prototype named WiSign with commercial WiFi equipment,which realizes word level and sentence level sign language recognition,and evaluates its performance in a real indoor environment.The results show that WiSign achieves satisfactory accuracy when recognizing ASL sentences that involves the movements of head,arms,hands and fingers.The research content and contribution of this thesis included the following three aspects:1.At the word-level sign language recognition section,we propose a segmentation method based on power spectral density(PSD)to capture the transitory pause between successive sign words and segment a sign sentence into isolated sign words.In this thesis,the similarity of high energy spectral values of the same motion is found,and a cross-correlation function is proposed to reduce the error caused by ”transition gesture”in time domain segmentation.This thesis combine the DBN and HMM with Gaussian mixture which is a method of feature extraction and classification of time and space information.This hybrid model extracts effective features under different operating environments at multiple levels.2.At the sentence-level sign language recognition section,the CCA-HAN framework is proposed to bypass time-domain segmentation,so that transition gestures do not need to be removed.Through the use of attention model,key sign language movement features are automatically extracted,and the semantic gap between the one-hot vector of sign language words and CSI data is narrowed,which improves the accuracy of sign language recognition.3.This thesis implement a prototype of WiSign with commercial WiFi devices and evaluate its performance in a real indoor environment.Experimental results showthat in different environments,at the word-level language recognition section(or sign language recognition at the sentence-level),the rate reaches up to 92% and 87%(95% and87%)with personalized model and 69%(65%)with the general model for thirty users,respectively.