Research On Lip-reading Based On HMM And Deep Learning

With the development of technology,life has been improved and electronic products have become part of daily life.Along with all these changes,there comes the higher and higher demands for electronic products,which leads to the arising of lip-reading technic.Lip-reading can be of great help in many circumstances,for instance,Automatic Speech Recognition(ASR)doesn't work very well when its surrounding is noisy;communication is always a terrible problem for patients with language disorders and the security of current privacy protection methods which are based on either password or fingerprint is decreasing day after day.Another example could be that even in the fields of military and homeland security,lip-reading helps to gain more information and fight crimes efficiently.However,as a newly-developed technic,although there're hundreds of thousands of approaches in all these fields related to it,lip-reading has lower recognition rate and more constraints,which in return makes itself to be just a theory studied in lab rather than a method applied in life.To deal with this problem,this paper firstly gives a brief analysis on face detection,lip location,feature extraction and lip-reading recognition;then on the basis of these theories,a lip-reading system based on Hidden Markov Model(HMM)and Deep Learning is proposed,along with an unlock method through lip-reading which is to be used on electronic products.The lip-reading system proposed in this paper builds an HMM model for each feature sequence,which will be trained before it can be used.To recognize a word,the system will calculate the likelihood of a certain HMM model for the given observation feature sequence.The features used in this system includes the height of outer lips,the height of inner lips and the width of mouth as well as the first derivatives of them three versus time.These six features can be calculated as long as the landmarks on the face are detected.Upon these,this paper takes advantage of both the Gaussian Mixture Model(GMM)and the Deep Belief Network(DBN)to model a viseme.Similarly,the unlock method through lip-reading uses the height of outer lips,the height of inner lips and the width of mouth as features,and GMM as the model of viseme.As the experimental results suggest,the lip-reading approach proposed in this paper works well,and its recognition rate meets the demands;on the other hand,the unlock method through lip-reading can reduce the probability of passwords stolen,which makes it much safer than the traditional password mechanism as well as the fingerprint-based methods and so on.