Multi Ship Fusion Tracking Based On Infrared And Visible Images

Vision based multi ship tracking technology is a key component of ship perception system and water monitoring system,which provides important information support for ship obstacle avoidance decision-making and maritime early warning.Most of the existing visual multi ship tracking technologies are single source image tracking technologies based on visible light or infrared.However,due to the limitations of image imaging mechanism,the visibility of visible image in sea fog,night,rainstorm and other environments is poor.Although infrared image can better adapt to low visibility environment,it lacks detailed information such as target color and texture.Therefore,multi ship tracking technology based on single source image is difficult to meet the needs of all-weather tracking;At the same time,the particularity of sea and sky environment also brings severe challenges to multi ship tracking,such as the large change of ship target scale and the decline of tracking accuracy caused by far and small targets.From the perspective of multi-modal information fusion,this paper uses the complementary advantages of infrared and visible images to achieve stable and accurate tracking of Multi-ship targets.Aiming at the special problems of marine scene,this paper studies the multi ship tracking algorithm based on visible light and infrared fusion based on depth network.The main work is as follows:(1)In view of the lack of multi ship fusion tracking data set based on infrared and visible light,the ship fusion data set DLF is constructed by using the ship video data collected by the experimental team from Qingdao sea area.The data set can meet the training and testing needs of ship detection and recognition algorithm and tracking algorithm based on deep learning.(2)Aiming at the unreliability of multi ship tracking based on single source image,a multi ship fusion tracking algorithm with adaptive weight(VTNet)is proposed based on the visible multi-target tracking algorithm Deep Affinity Network(DAN).The algorithm modifies the DAN into a feature level fusion tracking framework by adding infrared feature extraction stream and feature fusion layer.It adopts the fusion strategy of adaptive weight based on L1-norm,and dynamically adjusts the proportion of infrared and visible feature fusion through the action degree weight graph.The experimental results on DLF dataset show that VTNet is better than the tracking algorithm based on single source information It can improve the adaptability to the marine environment.(3)Aiming at the problem that the feature fusion granularity of VTNet algorithm is insufficient and can not adapt to the large-scale difference of ship targets and far and small targets,a fusion tracking algorithm based on attention mechanism(FA_DAN)is proposed in this paper.The algorithm adopts a feature fusion network based on attention machine on the basis of VTNet,proposed in modal The fusion strategy of thinning features in three dimensions(infrared and visible images),channel and space can obtain more recognizable fusion features;the robustness of the network to multi-scale target tracking is enhanced by embedding multi-scale feature accumulation module.The ablation experiment on DLF dataset proves the effectiveness of FA_DAN’s attention fusion strategy and multi-scale feature accumulation module;Environmental adaptability experiment verified FA_DAN algorithm has good robustness to sea fog weather,multi-scale targets and far and small targets.(4)In view of the poor real-time performance of the above detection-based tracking algorithm,based on FairMOT algorithm,this paper proposes a multi ship fusion tracking algorithm FDT_Net for joint detection.FDT_Net uses the way that detection tasks and tracking tasks share fusion features,which reduces the loss of feature extraction time and improves the tracking speed;FDA_Net introduces the fusion layer and adopts the feature level fusion of infrared and visible images is realized based on ZCA and L1 norm;CBAM module is embedded in the fusion feature extraction network to optimize the fusion features and improve the tracking accuracy of the target tracking algorithm.The experimental results on DLF dataset show that the average tracking speed of FDT_Net is 24.5 F/S,which meets the requirements of real-time;The MOTA index of the algorithm reaches 72.5%.FDT_Net realizes the balance between tracking accuracy and tracking speed.