Research On Driver Fatigue Detection Based On Machine Vision And Deep Learning

Fatigue detection researches based on physiological response,mostly employ machine vision technologies to detect drivers’ face,eye and head related features,but there are some technical limitations mainly in the following aspects: the performance of conventional machine vision methods is easily affected by illumination,pose and occlusion.Besides,precisions of detecting eyelids,pupils and head pose depend on the efficiency of algorithms,which should further improve real-time performance and accuracy.This article takes facial information processing methods based on deep learning as principal tasks,which could improve performance of real-driving fatigue detection.This work focuses on eye landmarks detection,3D head pose estimation and fatigue detection models.The main researches of this article are summarized as follows:(1)An eye landmarks detection model via two-level cascaded CNNs with multi-task learning is presented.First,an eye landmarks detection dataset is built and analyzed,which is named as OCE dataset.Then we introduce principles and implementations of multi-task learning and cascaded structure for eye landmark detection in details.Besides,we design two-level cascaded convolutional neural networks to detect eye landmarks coarse to fine,which is able to balance the size of the input image and the elimination of background noise.Network at the first level utilizes hard parameter sharing mechanisms to achieve multi-task learning.It takes eye state estimation as an auxiliary task and a regularizer to extract the more intrinsic features of eye regions.Finally,experimental results on OCE testing dataset show that accuracy of models with cascaded structure and multi-task learning is improved by 27.5% and 13.4% compared to benchmarks,respectively.Further experiments are tested on UBIRIS V1.0,MMU V1.0 and MICHE-I and their results demonstrate better effectiveness and generalization of our model.(2)A weakly supervised eye landmarks detection algorithm is proposed.This method can detect eye landmarks from severe occluded or local view of facial images.First,we introduce the special format data of WSS dataset,which contains supervised and weakly supervised samples.Then we analyze eye landmarks detection in the paradigm of weakly supervised learning with faster R-CNN and recurrent learning module.Faster R-CNN is trained end-to-end using supervised and weakly supervised samples,which is competent to detect bounding-box of facial components and initial positions of the eye.Recurrent learning module utilizes the initial eye shape to merge supervision information from different stages for eye landmarks refinement.Experimental results on WSS testing dataset show that,faster R-CNN with two kinds of samples improves by 17.7% compared to the model with only supervised samples.Besides,recurrent learning module improves 24.88% and 11.0% compared with the benchmark model and cascaded model,respectively.Futher experiments on LFPW and 300 W testing dataset demonstrate acceptable performance of our method on facial landmarks detection.Finally,two experiments on UBIRIS.v2 and MICHE datasets prove the improvements of eye localization.(3)A head pose estimation method using two-stage ensembles with average top-k regression is proposed.We give in-depth study about the influence of angle interval to head pose estimation algorithm,which is based on classification-based methods.Then we select two angle intervals and theoretically derive the average top-k regression for head pose estimation.Additional experiments show that the influence of different k values for the performance of head pose estimation.Then we derive a combined loss function with parameter learning strategy for classification and regression tasks,which can learn task-dependent weights automatically from training data.The strategy avoids complicatedly assigning hype-parameters manually and improves the accuracy of head pose estimation.Finally,experimental results that MAE on AFLW2000 and BIWI datasets are 5.28° and 4.29°,respectively,demonstrate the effectiveness and generalization of the proposed method.(4)Two fatigue detection methods based on statistics and temporal LBP-3DCNN are proposed.First,we analyze samples characteristics of fatigue dataset.Then we extract eye-moved features using statistics method and 3D head pose to build parameter space of fatigue features.Based on the feature space,we design shallow networks fatigue classifier,which gets the accuracy of 85.91% on testing set of fatigue dataset.This article further attempts to study fatigue detection model based on temporal 3D LBP-CNN.The model takes multi-frame sequences as input and extract spatial appearance and temporal motion features using temporal LBP texture features and 3D convolution module.This method builds deep LBP-3DCNN by end-to-end learning,and integrates the fused feature of three channel networks to output the drowsiness level of drivers.The accuracy on testing set of fatigue dataset is89.54%,which is improved by 4.23% compared with the method based on statistics.Experimental results demonstrate the effectiveness of our method.