Research On Multi-modal Fusion Fatigue Driving Detection Method Based On Heart Rate And Eyelid Features

Fatigue driving will lead to a decrease in the drivers reaction speed,thus increasing the possibility of traffic accidents.Detection of driver fatigue could prevent many accidents.Therefore,it is very important to design an algorithm to detect fatigue driving.Existing fatigue detection methods usually capture drivers behaviors such as blinking,yawning and dozing to identify whether a driver is tired.The advantage of this method is that the data is captured by the camera without disturbing the driver.However,there are also many studies that use physiological signals as an important fatigue detection data.If we can simultaneously integrate physiological signals and driver behavior data for fatigue detection,the accuracy of the whole system will be improved.Electrodes attached to the drivers body to collect physiological signals can interfere with driving.There is a contradiction between the non-intrusive nature of fatigue detection based on driver behavior and the intrusive nature of fatigue detection based on physiological signals.In order to solve this contradiction,this thesis proposes a method to extract driver behavior and heart rate signals from RGB video,which can be used for driver multi-mode fusion fatigue detection.The driver behavior characteristic used in this thesis is the PERCLOS value.This provides a new solution for non-intrusively integrating driver behavior and physiological signals for fatigue detection.The main work of this thesis is as follows:1)An unsupervised learning method is proposed to approximate the PERCLOS value based on the RGB video;our idea is to first obtain the eye area through the face detection algorithm.Then use the OTSU algorithm to binarize the eye area image.The iris is easy to adhere to the eyelid area in the binary image obtained by the OTSU binarization.In order to segment the eye area more finely,we use K-means clustering algorithm to segment the eye RGB image.Then calculate the OTSU algorithm to obtain the Io U value of the binary image and the different label segmentation image obtained by K-means to locate the area where the iris is located.The final PERCLOS calculation method is to obtain the PERCLOS value when the eyelid covers the iris area for more than 50% of the time period.Finally,a one-dimensional convolutional neural network is used to classify the PERCLOS signals in a period of time.2)We designed a vision-based heart rate method for fatigue detection;our core idea is to first obtain the remote photoplethysmography signal according to the RGB video,and then obtain the heart rate according to the remote photoplethysmography signal.In this chapter,we first introduce the common algorithms for remote photoplethysmography acquisition.And introduced how we obtain remote photoplethysmography signal.The frequency of the remote photoplethysmography(r PPG)signal is then calculated to get the heart rate.Finally,a onedimensional convolutional neural network is used to detect fatigue based on the heart rate.3)Finally,a multi-modal fusion strategy was designed to fuse heart rate and PERCLOS values for fatigue detection.In this paper,two independent fatigue detection sub-models were obtained according to the value of PERCLOS and heart rate.Then multi-mode fusion fatigue detection is realized by the weighted average of the output of the two sub-models.Experiments are designed to verify the performance of the algorithm.