Researches On Lip Reading Using The Color And Depth Image

Lip-reading technology is a new interdisciplinary research area, covering severalkey technologies of image processing, pattern recognition and computer vision. With itsvisual advantages, it is widely used in assistant rehabilitation of the deaf, assistant voicerecognition, gesture recognition, and personal identity authentication and so on. Basedon the summary of present research results home and abroad, the key modules on lipreading techniques are intensive studied in this project. The major works are listed asfollows:(1)The small colored and depth image database is constructed because of the factthat corresponding database between color picture and depth image is needed due to theparticularity of research direction of this project, however, at present there is no such amature database, and the preliminary video segmentation is completed and key frameimages are achieved.(2) The related research of location of lip region is conducted. The preciselocalization with the aid of the method of zone location of lip based on the facedetection and relative structure of facial features is carried on and through the massiveexperiments, the optimum parameter of lip in the face region distribution, comparedwith that of nose, is proposed. The experiment result indicated that the parameter whichthis project has put forward can demarcate the lip region precisely and eliminate otherdisturbances as far as possible.(3) In terms of lip division technology research, using the color and depth imagelip division methods are presented in the project. Regarding the color image, thesuggestion that better division threshold value using the Otsu method could be obtainedin the research of threshold value choice is raised based on the lip extraction methodusing a-plane in Lab color space. Regarding the depth image, lip region of depth imageis obtained through mapping of the color image and the division of the depth image lipzone using the K-means cluster is adopted. The research of union and denoisingmethods of these two images is conducted. The experiment proves that the lip and the non-lip picture elements could be separated precisely and effectively with the help ofthe lip division method of color and depth image.(4) In terms of lip feature extraction, the feature extraction method based on theprojection and the deformable template is utilized. Firstly, the initialization spot of thedeformable template is found precisely using the projection, and then makes thedeformable template iterate nearby the initialization spot, which reduces the iterativetimes massively and enhances the matching speed of invariable template. Moreover, interms of recognition method, the classified experiment using the SVM classificationmethod is carried on, which indicates that the method adopted in the thesis couldenhance the recognition rate of the lip-reading system.