Lip Reading And User Authentication Through Ultrasonic Sensing Based On Mobile Devices

With the enhanced computing capability on mobile devices,the speech recogni-tion based service is embedded in lots of applications,and its performance has been greatly improved recently.Traditional speech recognition records acoustic signals or visual images to analyze what people are saying or who is talking.However,the acous-tic based scheme is easily affected by the environmental noises,and can not be used in some public places where people need to be quite to avoid disturbing others.Besides,it is not suitable for people with speaking or hearing difficulties.The visual-based method is sensitive to light conditions and capturing angles.Moreover,the high com-putation resources consumption may prevent its application in mobile devices.Thus,it is necessary to provide a new human-computer interaction to assist speech recogni-tion.Recently,ultrasonic signals have been widely used to sense and analyze human's activities like walking,sitting down,waving,breathing and so on.Many works lever-age the reflected signal's variation caused by body movements to estimate the object's activities.And mouth movement is a type of fine-grained activity.This paper novelly exploits ultrasonic signals to track mouth movements.The main idea can be listed as follows:Firstly,we transmit near-ultrasonic frequency(16-22KHz)signals from the commercial mobile device's speaker.The variations of mouth movements like speed,distance,direction and so on,will affect signal's frequency,en-velope,phase and so on.Such variations are captured by the device's microphone.Then,we extract fine-grained features of speech and speaker.For lip reading system,this paper detects the Doppler Effect to quantify the correlation between frequency variations and 12 basic mouth motions.Then,apply a language model to realize con-tinuous lip reading,which combines pronunciation rules and context knowledge.For user authentication system,this work extracts three types of features from signals'en-velope as users'ID.Next,we build a classifier to classify each new input as legitimate or illegitimate.Such design has the following advantages:First,it solves above problems of speech recognition.Second,ultrasonic based solution will not be affected by environ=mental noises or light condition.Our framework is able to be deployed in commercial mobile phones since most of mobile phones are equipped with standard speakers and microphones.The experimental results show that above two systems reach high accu-racy.The lip reading system can identify 12 basic mouth motions up to 95%accuracy,and recognize short sentences up to 74.8%accuracy.The user authentication system realizes 93%TNR and 83.1%TPR with 9 training samples.