Research On Lip Reading Algorithms

Human’s perception of language is multichannel-even people with normal hearing use information such as lip movement and facial expression unconsciously to assist with hearing when talk with others. The visual information becomes more important to hearing impaired person or in noise environment. The computer lip reading technology is arising under such a background and deals with the fields of computer vision, human-computer interaction, image processing, pattern recognition and so on. It has been widely used as complement to improve the speech recognition and also been used in public safety, biometric identification, lip-motion synthesis and so on.This paper focus on lip reading system based on visual channel only for Chinese isolated words and studies from3aspects:the lip area detection, feature extraction and lip reading recognition. The main works and innovative points are as follows:(1) This paper summarizes the existing algorithms of lip area detection and introduces the Viola-Jones detection method in detail. The Viola-Jones method proposes a detector which based on Haar-like feature and Adaboost algorithm. It defines bunch of Haar rectangle features and trains a week classifier for each feature using Adaboost with the training dataset. We use this method to detect faces and lips in this paper and achieve an accuracy rate of98%.(2) Feature extraction is an important part of the lip reading system. This paper compares the advantages and disadvantages of the existing3sorts of methods and focus on the pixel based methods. We proposes a4-stage cascade feature extraction method which includes the DCT-based and DWT-based image transformation, scanning of coefficients, PCA-based dimensional reduction and K-means based vector quantification. The result shows that the recognition rate using the4-stage feature extraction method can be improved about10%compared to the use of DCT or DWT.(3) This paper uses the HMM to do the training and recognition. We introduce the working principle of HMM and discusse the solution of the problems that would face in actual application of HMM. The process of applying HMM in training and recognition is also given.(4) The database for training is the foundation condition of lip reading. We do some research on the available databases and analysis their corpus and parameters. Then we establish our own database which includes38common used Chinese words as corpus and5persons who announced each word for10times. The videos are processed after captured and the lip area of each frame of image is extracted. This paper also builds a lip reading system based on C++and MFC which realized real time recognition of Chinese isolated words.