| Target tracking is a hot topic in the field of computer vision.It has a wide range of applications in intelligent visual navigation,intelligent transportation,battlefield reconnaissance,security monitoring,automated production and other fields.It has a major contribution to promoting social and economic development and promoting social progress.Therefore,in recent years,this field has attracted more and more researchers’ attention and many breakthroughs have been made.However,it still faces many challenges,such as the difficulty of effective target tracking in low light,rainy weather,and occlusion.This is due to the fact that the feature of the visible light pictures are difficult to effectively express the target feature in these situations,and it is difficult to solve the problem just by designing the new network architecture or changing the network module.For this reason,RGB-T(RGB-Thermal)tracking came into being.This direction has also attracted more and more attention in recent years.It uses different sensing devices to obtain thermal pictures and visible pictures.Thermal pictures are not sensitive to light,but have good penetrability for smoke and occlusion.It provides powerful information supplement for visible light information,but it lacks details and texture information,which can be supplemented by visible light information.The combination of the two modals can make tracking more accurate and robust.This paper mainly studies the feature extraction and fusion problem in RGB-T fusion tracking under the framework of deep learning,exploring how to effectively fuse the two modal features and extract effective two model features to improve the accuracy of the existing RGB-T fusion network.The main research results are as follows(1)Aiming at the challenges that current object tracking is difficult to adapt to,like low light,motion blur,fast moving targets etc.,an RGB-T dual-modal interactive fusion tracking network under spatial channel attention,named Spatial Channel Attention Interaction Network,is proposed to obtain excellent tracking performance by improving the feature fusion part.First,the infrared and RGB images are extracted through the backbone(VGG-M)three-layer convalution to extract layered features which are normalized to the same resolution via dimension reduction.The three-layer features are cascaded to form each modal feature.Then the features are sent to the designed spatial channel self-attention module and the cross-module interactive attention module,which lead network focus on global spatial features and high-response channels and therefore,improve the complementarity of the dual-modal information.The interacted features of the dual-modal are cascaded for the fusion and finally sent to three fully connected layers to complete the target tracking.The experimental results on the GTOT dataset show that the network improves PR and SR by7.8% and 7.7% compared with the baseline network.(2)We improve the RGB-T tracking network for the direction of feature extraction,and propose a Multi-modal Multi-scale Patch Feature Network.At present,most of the RGB-T tracking frameworks focus on modal fusion which is performed by calculating weights.There are few articles focusing on improving feature extractio.Inspired by the patch division operation in transformer and the multi-scale parallel extraction of Inception Net,a multi-modal multi-scale patch fusion network is proposed.Firstly,multi-scale patch division is performed on the visible and thermal images.Then the same scale patches are extracted by using a convolutional network with parameters shared between modalities.Secondly,the modal patch features are obtained by fusing the same modality and different scale patch features.The hierarchical feature extraction network is used to extract the modal patch features,and finally the extracted visible and thermal features are contacted and sent to the fully connected tracking module which is the same as that of the MDNet to realize the target foreground and background recognition,and then complete the tracking.The experimental results on the GTOT dataset show that the network improves PR and SR by 7.8% and 8.0% compared with the baseline network.(3)On the basis of improving the Multi-modal Multi-scale Patch Feature Network in the feature extraction part,we propose a cross-modal information interaction method suitable for patch feature.A Cross-modal Patch Information Interaction Network is proposed,which is different from the conventional cross-modal fusion in this field that mostly uses the final modal features before cascading for cross-modal information interaction fusion.After multi-scale patch feature extraction,a cross-modal patch feature fusion module is added,and by calculating the interaction response map of the same scale features of different modalities,the cross-modal information can be obtained after the multi-scale patch feature extraction.During training,the double-modal features extracted from the hierarchical features are output,and the cross-modal orthogonal regularization loss is calculated to make the double-modal features more complementary and different to perform effective tracking.The experimental results on the GTOT dataset show that the network improves PR and SR by 6.9% and 9.4%compared with the baseline network. |
