Recognition Of Visual-related Non-driving Activities Based On Computer Vision

For the level 3 autonomous driving,according to the SAE International Automation Levels definition(J3016),the identification of non-driving activities(NDAs)that the driver is engaging with is crucial to design an intelligent take-over interface.Most of the existing literature pay attention to the take-over strategy with associated Human-Machine Interaction design.This paper proposes a dual-camera-based non-driving activity recognition which is low-cost,non-invasive,and can effectively identify five common non-driving activities that are driver-led by the vision,that is,using a mobile phone,using a laptop,reading,using a tablet and interact with the center console.First,the driver ’s gaze is estimated based on the Open Face algorithm.Eight feature parameters related to the driver ’s head and eyes are extracted,and mapping the driver’s gaze through the VNRX(Volterra Non-linear Regressive with e Xogenous)model,which is used to calculate the coordinates of the driver’s sight point.The effectiveness of the model is verified through three sets of experiments.The errors of this system in the X and Y directions are 7.80 ± 5.99 pixel and 4.64 ± 3.47 pixel for in-vehicle experiment.Then use the Mask R-CNN algorithm to perform pixel-level positioning on the target objects involved in non-driving activities.Based on the Py Torch deep learning platform,the Detectron2 detector containing the Mask R-CNN model is established.According to the annotation format of the MS-COCO dataset,a dedicated dataset of 250 images is built(200 for training and 50 for testing).The backbone network is Res Net101-FPN and model is trained and tested the on a computer with NVIDIA Quadro P6000 graphics card with 24 GB DDR5X memory.Finally,the non-driving activity classifier is used to identify and classify the non-driving activities,and the recognition effect is visually displayed in the form of heat map visualization.A novel gaze-based region of interest(ROI)selection module is introduced and contributes about 30% improvement of average success rate and about 60% reduction of average processing time compared to the results without the ROI module implemented.This framework has been successfully demonstrated to identify five types of NDA with an average success rate of 82.55%.The research in this paper provides a novel idea for the driver’s non-driving activities recognition,which will have wide applications on activities identification.The driver’s non-driving activities recognition could potentially be applied to evaluate the driver’s attention level.