The Key Techniques In Lipreading

Lipreading is a novel approach for Human-Computer Nature Interaction and Biometrics Identification. This paper concentrates on the feature extraction and language model in only visual channel lipreading.In the aspect of lip detection, adaptive chromatic filter model is introduced, which makes the system adapt to different lip color, illumination and hue of video camera, so that enhances the robust of the system in real world. Also we introduce the Mean-Shift algorithm based on the chromatic feature to segment lip pixels form scenes.Feature extraction plays an important role in lipreading, which is to get low-dimension, low-redundancy and representative features. This paper focuses on the pixel based feature extraction method. A three-stage cascade visual front end is proposed. The first stage is corresponding transform to be performed over the image, the second stage is to reduce the dimensions of the transformed image, in the third stage all feature vectors are normalized into a uniform scale. We apply KL to reduce the dimension of DCT and Gabor transformed data called DCT-KL and Gabor-KL, which can improve the recognition accuracy by 10% compared with the manually-selected features.In the aspect of lipreading recognition, HMM is used to train and recognize 200 sentences including 96 syllables. Some detail problems are discussed when built a calculated HMM model specially suited for lipreading recognition.Language Model (LM) is proposed as the last module in lipreading which has direct effect on the recognition accuracy. The LM has two applications in lipreading: First is to assist the recognition process, we combine the HMM probability with LM statistic, which can improve the recognition accuracy by 5%. Second is to carry out the Chinese Syllables-to-Words algorithm, the Chinese word recognition accuracy can achieve 70% based on syllable accuracy 82.4%.