Design And Implementation Of Pedestrian Liveness Detection System Based On Mmwave Radar And Computer Vision Fusion

Advanced driver assistance systems perform pedestrian detection by introducing sensors such as camera and radar and realize automatic planning of driving strategies and take active safety measures such as deceleration and avoidance to reduce the risk of pedestrian collision.Nevertheless,current related work still has certain shortcomings in detecting living pedestrian targets in complex road environments.First of all,there are living pedestrians and roadside advertisements printed with portraits in the road environment.However,existing studies based on computer vision usually carry out pedestrian detection based on image feature extraction or deep learning,which are prone to interference from portrait advertisements and unable to accurately detect living pedestrians.Secondly,existing radar-based target detection methods can detect targets within their working range.However,due to the sparsity of radar point clouds and the angular resolution significantly lower than the image,radar cannot capture the appearance of the target as the clue of classification.Finally,although target detection methods based on multi-sensor fusion improve detection performance by fusing the advantages of sensors,the shortcomings of sensors in living pedestrian detection are not compensated by multi-sensor fusion,and computer vision still cannot exclude the interference of portrait advertisements,and radar still has difficulty in capturing the appearance of the target.To this end,this thesis constructs pedestrian liveness detection system based on mmWave radar and computer vision fusion,extracts radar cross section feature from radar data oriented to living pedestrians,constructs radar pixel image data structure based on joint calibration,and achieves pedestrian liveness detection through mmWave radar and computer vision feature fusion model.The specific research work of this thesis mainly includes:(1)This thesis proposes a mmWave radar feature extraction mechanism for living pedestrians.Based on the principle of mmWave radar and the characteristics of radar data,this thesis constructs a mmWave radar signal data preprocessing pipeline to obtain the 3D point cloud of effective targets from the viewpoint of mmWave radar.For the characteristics of living pedestrians in road environment,this thesis proposes a radar cross section feature extraction mechanism based on the energy calibration of mmWave radar signal.The effectiveness of the radar cross section feature extraction mechanism and the rationality of using radar cross section feature to distinguish living pedestrians from portrait advertisements are demonstrated through experiments.(2)This thesis proposes fusion mechanism of features of mmWave radar and computer vision.Based on the spatial joint calibration of mmWave radar and computer vision,this thesis converts mmWave radar data and vision data under the same coordinate system,and establishes a radar pixel image data structure expressing mmWave radar features.This thesis constructs a feature extraction module of mmWave radar and computer vision data based on Darknet53 network.A feature fusion module is designed based on the attention mechanism to achieve the features fusion of vision image and radar pixel image,and the detection results are predicted from the fusion features.(3)This thesis designs and implements a pedestrian liveness detection system based on mmWave radar and computer vision fusion.This thesis investigates the performance of this system on a self-collected dataset and verifies the effectiveness of each mechanism proposed in this system by comparing the performance with existing work as well as the performance of this system after deleting the mechanism.In summary,this thesis introduces radar cross section feature into mmWave radar signal processing for living pedestrians,and designs and implements a pedestrian liveness detection system based on mmWave radar and computer vision by constructing a feature fusion model of mmWave radar and computer vision.The research results will promote the application of object detection in intelligent transportation and provide support for enhancing the environment perception capability of vehicle advanced driver assistance system.