Lipreading Technology Based On Lip Motion Feature

Human speech perception is inherently a multimodal process. Visual information in the speaker's mouth region is also an important speech information source except for acoustic signal source. Visual speech has many potential applications and has motivated significant interest in automatic recognition of visual speech, formally known as automatic lipreading, or speechreading.The paper gives a comprehensive survey and analysis of the existing lipreading technique, and describes the methods of current lipreading feature extraction. By systematically analyzing of relevant algorithms, we present some novel algorithms for lipreading, such as lip detection, feature extraction and lipreading recognition. In addition, a prototype system of lipreading is designed and implemented. The detail is described as follows:First, the skin model and the geometrical feature are used to detect the face. Further use a lip examination algorithm of multistage structure to detect the lip. After examining the face, we use the Fisher transformation aglorithm to enhance the lip region; then use Otsu to generate the binary image, completing the thick division, then combine the YIQ lip color model to carry on the lip examination, compieting a most superior division. Finally use the gray-scale projection to locate the lip. The whole process has used the chromatic information and the apriori knowledge fully. Using the division result to initialize the contour parameter by this method can enhance the speed and the accuracy in contour extraction greatly.In the aspect of lip tracking and feature extraction, a new approach based on improved snake algorithm is proposed to extract the lip contour. This method is to develop a new approach to generate an initial contour automatically first; Then a new external force is redesigned basing on the GVF-snake. We use the color difference algorithm to get the edge grads of interest. In addition, we normalize the GVF field and improve the smooth factor. Then combine the optical-flow and snake algorithm to track the serial images. To extract the lip motion feature, we propose a lip motion feature extraction algorithm based on motion vector of feature points between frames. We extract suitable feature points and study the lip motion rule between the consecutive frames, extracting the features suitable for lipreading.In the aspect of lipreading, the BP neural network algorithm for lipreading is proposed. With the combination of additional momentum and adaptive speed, we get a rapid BP neural network algorithm. The results indicate that our method is robust and successful in the conditions of dependent subject.The test result indicated that the lip recognition algorithm based on lip motion proposed by this article is effective. It can adapt to the changes of illumination and lip color in a certain degree, considering the change of lip contour feature fully when pronunce. The system can complete the recognition under the video environment and real-time pronunciation in the permission delay.