| Vision is an important dimension for human perception of external information.Lipreading aims to recognize the language content of speech by observing lip movements.It’s an important way to interact with information in noisy and silent environments.In recent years,with the increasing demand for visual information interaction applications,lipreading has gradually become the focus of attention in the field of artificial intelligence.Lipreading models have roughly gone through the evolution from pattern recognition to traditional machine learning and then deep learning.At present,lipreading research is still in its infancy.Most researchers use deep learning methods to recognize lip deformation through pattern recognition and matching.The key technologies,such as lip feature extraction,time-series modeling,and model lightweight,still have many challenges in real-world applications.Distinguishing lip deformation to improve the accuracy of lipreading is an important research direction for achieving efficient expression of lip movement rules.In this paper,we combine the characteristics of lipreading to study the efficient expression algorithm of lip movement rules,which is of great significance to improving the lipreading research system and promoting the rapid development of lipreading research.With the continuous development of deep learning technology and theory,the accuracy of lipreading has been greatly improved.However,since lipreading is a comprehensive topic that combines computer vision and natural language processing,there are still the following problems that need to be solved:the existing models do not fully consider the small and difficult-to-distinguish characteristics of lip movements;the selection of lip feature representation units is blind,lacking of reasonable analysis methods;lip movements contain relatively little language information,and the use of audio information is not sufficient.To solve the above problems,we combine the laws of lip movement,the basic structure of Chinese and the principles of speech production.Carry out research in three aspects:lipreading model based on the fine-grained global synergy characteristics of lips,lipreading recognition unit applicability analysis and model verification based on information entropy,and modal amplification lipreading based on audio feature reconstruction.The main research contents are summarized as follows:1)To solve the problem that existing lipreading models do not adequately consider the small and difficult-to-distinguish characteristics of lip movements,we proposed a lipreading model based on the fine-grained global synergy characteristics of lips.Through the analysis of the liprelated muscle tissue structure and the comparison with visual recognition tasks in other fields,this section concludes that lip movements have the characteristics of fine-grained global synergy.At the same time,by learning the correlation between local features,the fine-grained correlation between different positions of local features,and the global synergy between local and global features,we constructed a feature extraction model that conforms to the rules of lip movement.Experimental results show that designing corresponding models based on the rules of lip movement can improve lipreading accuracy.Compared with the baseline model,after integrating the fine-grained global synergy of the lips,the improved model performance increased by 6.96%on the ICSLR dataset.2)To address the issue of blindly selecting lip feature representation units during lipreading research,we proposed a method for analyzing the applicability of lip recognition unit based on information entropy and a model verification method.In this study,by analyzing the basic structure of Chinese,we constructed five recognition units suitable for Chinese lipreading.The phoneme-viseme mapping table is established in a datadriven manner,and the viseme is used as the lip feature representation unit.At the same time,we presented the applicability analysis results of the Chinese lip recognition unit based on the information entropy theory,and we constructed a Chinese lipreading model that fuses multi-feature spaces to verify it.Experimental results show that the information entropy theory can be used to analyze the applicability of the lip feature representation unit.And after optimizing the recognition unit and model,the performance on ICSLR data increased by 2.15%.3)To tackle the problem that the visual articulators have weak ability to express language during lipreading,we proposed a modal amplification lipreading model based on audio feature reconstruction.By analyzing the pronunciation principles,we design a novel research idea to assist in enhancing lip features by reconstructing audio features.Considering the light weighting problem during model building process,we constructed a lightweight lip feature extraction based on temporal shift module and a lightweight audio feature reconstruction model based on improved convolutional encoding and decoding structure.Furthermore,to avoid the impact of invalid features on model performance,we introduced an attention mechanism into the audio feature reconstruction process,and filtered and fused the reconstructed audio features and lip features.Experimental results demonstrated that compared with the single-modal structure,the model performance after modal expansion is improved by 5.40%and 3.25%on the ICSLR and CMLR datasets respectively. |
