Research On Hidden Markov Model Based Computer Lipreading

Lip-reading is implemented by computer to infer speaker’s words byanalyzing his lip image. This technology has attractive prospect in speechrecognition, human computer interaction, deaf aid and many other fields.Computer lip-reading mainly has three parts, which include lips location, featureextraction and movement recognition. This thesis has focused on these.This thesis has established a lip-reading corpus, which contains180samplesfrom three speakers. This process has three steps, audio-video recording, videocropping and parsing. In the final step, a frame capture software based onDirectShow has been developed to obtain images.This thesis has used template matching method to locate lips. A facialgrayscale template has been constructed. The template is symmetric andcomposed of13regions, which have areas coinciding with facial organs.Grayscale relation between regions reflects the status of facial brightness. Whendetecting, the template slides on image until matches. In this case, lips can belocated according to the template structure. Experiments have shown that thismethod is stable and effective.In this thesis, a key point based mouth feature extraction algorithm has beenproposed. Two corners of mouth, two sharp points of the upper lip, as well as twomidpoints of lips are selected as key points. The algorithm first detects thecorners of mouth by local minimum grayscale searching, then, locates the keypoints of upper lip in the hybrid gradient field by jumping snake method.This thesis has achieved good recognition result of lip movement based onHMM（Hidden Markov Model），and proposed a new sentence level lip-readingmethod by fusing word model with bi-gram sentence network. Compared with traditional isolated recognition, the proposed method can reduces the amount ofHMM training, especially under the situation of large vocabularies.