Design And Implementation Of Image Fusion-Based Hand Tracking System

With the development of intelligent car cockpits,real-time motion tracking of passengers’ hands to achieve perceptual interaction with interior lights has become a hot demand in the industry.However,poor cockpit illumination will lead to low image acquisition quality,and missing detection of small targets will lead to tracking loss and tracking discontinuous.Aiming at the above two problems,this paper adopts the method of image fusion of visible light image and near-infrared image in the cockpit,combined with the improved YOLOv4-Tiny detector and Deep SORT tracker,and proposes an image fusion-based hand tracking system.The system can effectively improve the quality of image acquisition under low illumination conditions and realize accurate tracking of hand small targets in the cockpit.The main contributions of this paper are as follows:(1)This paper designs an image fusion method of visible light image and nearinfrared images in the cockpit to improve the quality of image acquisition under low illumination conditions.Based on the SURF feature points,the near-infrared image and the visible light image are registered,and then the registered near-infrared image are fused with the visible light image through wavelet transform in the YUV color channel.This method effectively enhances the contour features of the target,and significantly improves the information entropy of the fused image.Under low illumination conditions,the entropy increase rate compared to visible light images reaches 72.7%,and the entropy increase rate compared to near-infrared images reaches 22.3%.(2)This paper designs a hand tracking algorithm based on improved YOLOv4-Tiny and Deep SORT to improve the performance of hand small target detection and tracking.Firstly,optimizing YOLOv4-Tiny to enhance the performance of hand small target detection,the optimization methods are: add shallow feature branches to the enhanced feature extraction network to introduce more position and detail information,design an improved PANet network with Depthwise Separable Convolution to enhance the ability to fuse features of different scales and compress the model volume,and also design an improved NMS method as a non-maximum suppression method for the prediction network.Secondly,optimizing Deep SORT to improve the performance of hand small target tracking,the standard Kalman filter of the motion estimation module in Deep SORT is extended to Cubature Kalman filter,making it more suitable for hand nonlinear motion application scenarios.The above improvement method improves the tracking accuracy and tracking precision of small hand targets by 13% and 8%respectively compared with the standard YOLOv4-Tiny+Deep SORT algorithm.(3)Made a self-made cockpit hand data set named Cockpit-Hand,and build a real automobile embedded environment to verify the algorithm of this paper.Due to the absence of public car cockpit hand datasets,a self-made cockpit hand dataset is realized by collecting and manual labelling 10,000 hand images under different illumination environments in the cockpit,which is used to train the algorithm.Then the algorithm is deployed on the embedded platform and a real vehicle cockpit environment is built to verify the algorithm.The experimental results show that in the real vehicle cockpit environment,the tracking accuracy and tracking precision of the system proposed in this paper reaches92% and 90% respectively.The real-time frame rate reaches 18.6fps,which meets the automotive embedded requirements.In addition,the hand tracking system can be combined with intelligent interactive reading lamp in the cockpit to create a diverse interactive scene of illumination tracking,light can track hands in real time when passengers are engaged in various cockpit hand movements,such as reading books and newspapers,searching,using laptop and so on.