Real-time Line-of-sight Estimation Method Based On Appearance

Line-of-sight estimation is one of the important research directions in the field of computer vision,and has been valued by many researchers in the fields of attention analysis,human-computer interaction,VR/AR and assisted driving.At present,there are two main methods of line-of-sight estimation: model-based and appearance-based.Model-based methods often require users to wear complex devices or infrared light sources to cooperate,which limits their application;The appearance-based gaze estimation method does not require wearing complex equipment,and the user experience is good,so it has gradually become the mainstream gaze estimation method.In this thesis,the appearance-based gaze estimation method is adopted,aiming at the problems of low accuracy,poor real-time performance and difficulty in adapting to multiple head poses of existing gaze estimation methods,taking the full-face image and head pose as the input of the shallow residual network fusing the attention mechanism of Sim AM,and the gaze direction as the output,and training and testing on the MPII gaze estimation dataset,the specific work is as follows:(1)Accurate and fast face detection in the appearance-based gaze estimation task is the primary task,this thesis selects the method based on HOG features combined with SVM for face detection,uses gradient boosting decision tree to locate68 key points of face on the basis of detecting face,and then combines the eye aspect ratio index to judge the blinking behavior,remove invalid video frame images,and provide more accurate input for subsequent gaze estimation.(2)Aiming at the problem of camera calibration and head pose estimation in line of sight estimation,10×10 black and white checkerboard calibration templates with a size of 28 mm × 28 mm were used to calibrate the camera;Then,combined with the obtained camera internal and external parameters,face key points and Pn P algorithm,the head pose is estimated and compared with the head pose collected by the MPU6050 gyroscope to verify the effectiveness of the head pose estimation algorithm proposed in this paper,and the head pose information is used as the input of the line of sight estimation network to solve the problem that the existing line of sight estimation algorithm limits the head pose or fails to make full use of the head pose information.(3)The shallow residual network fusing the Sim AM attention-free attention mechanism based on energy function is used as the backbone network for feature extraction.The Sim AM attention mechanism is introduced to directly derive the 3D attention weight for the feature map without increasing the amount of network parameters,so that the network can learn more clearly the importance of different areas of the face to the estimation of gaze,and enhance the flexibility and effectiveness of the network.The results of the cross-validation experiment on the MPII line-of-sight estimation dataset show that compared with the two monocular image line-of-sight estimation methods based on Le Net and Gaze Net networks,the average angular error decreases by 1.98° and 1.41°,respectively,and the average angular error decreases by 0.47° compared with the full-face gaze estimation method of Alex Net with spatial weights.The Sim AM attention mechanism heatmap distribution indicates that the eye area is the focus of attention in any case;Finally,the performance of the gaze estimation method in this paper is tested,and the final gaze estimation effect is demonstrated.