Nighttime Thermal Infrared Image Colorization Methods Based On Visual Cognitive Mechanisms

All-weather scene perception is crucial for various applications such as autonomous driving,intelligent surveillance,and rescue robots.Among different visual acquisition methods for night scenes,thermal infrared cameras are important because they are stable in imaging glare and low-light environments.However,thermal infrared imaging results are typically presented in grayscale images and suffer from low contrast,blurred boundaries,and poor detail representation.These limitations not only impact the perception of human observers but also hinder the transfer of existing scene perception models based on visible images.Therefore,nighttime thermal infrared image colorization is essential to facilitate the application of related computer vision methods to all-weather scenes with multimodal data.The goal of this task is to colorize nighttime thermal infrared images into daytime color images with consistent content.Due to the lack of paired image labels,existing colorization methods usually suffer from poor semantic consistency.In comparison to deep neural networks,the human visual system has notable advantages,including stability of cross-modal reasoning,generalization of small-sample learning,and systematicity of learning strategy.Inspired by this,this dissertation focuses on nighttime thermal infrared image colorization methods based on visual cognitive mechanisms.Specifically,this dissertation first divides the implementation process of colorization into three stages: encoding,mapping and presentation,which correspond to feature encoding of thermal infrared images,cross-domain mapping of semantic features and appearance presentation of colorization,respectively.These three stages serve as the main line of research.Then,based on visual attention,associative memory,and feedback learning mechanisms of brain,this dissertation delves into the task of colorizing nighttime thermal infrared images of traffic scenes,with a focus on reducing neighborhood entanglement(i.e.,semantic confusion of neighboring regions)during feature encoding,improving the semantic consistency of cross-domain mapping,and enhancing the appearance presentation of small objects.These efforts aim to significantly enhance the performance of thermal infrared image colorization.The main research contents and contributions of this dissertation include:1.A new image structure consistency metric that uses edge alignment is proposed to lay the foundation for assessing the multi-granularity consistency of colorization models.This dissertation aims to address the lack of effective and reasonable quantitative metrics for evaluating image-to-image translation performance.Firstly,pixel-level semantic annotation is carried out on three frequently used thermal infrared datasets.Next,a new evaluation metric based on edge consistency before and after colorization is developed to facilitate relevant model design and effective evaluation in the field of thermal infrared image translation.2.A visual attention-based colorization method is designed to reduce the neighborhood entanglement problem in the colorization encoding stage and enhance the representation of complex scenes.Inspired by the visual attention mechanism,a top-down guided attention module is proposed to guide the low-level attention with high-level features.The module integrates contextual information to reduce neighborhood entanglement during thermal infrared image encoding.The experimental results demonstrate that this method not only learns spatially separated hierarchical attention,but also enhances the colorization of challenging scenes.3.An associative memory-based colorization method is designed to reduce the semantic distortion problem in the colorization mapping stage and improve the semantic consistency of small-sample category translation.Inspired by the associative memory mechanism,a memory-guided sample selection strategy is proposed.The strategy first performs online memory for pseudo-labels of thermal infrared images,and then associates thermal infrared images with similar distribution for the input visible images.Thereafter,the method designs an adaptive collaborative attention loss to enhance feature similarity of cross-domain images over small sample categories.Experimental results show that the proposed method not only reduces the semantic distortion problem in the mapping stage,but also significantly outperforms other methods in terms of semantic consistency for small sample categories.4.A feedback-based colorization method is designed to reduce the appearance distortion problem in the colorization presentation stage and enhance the presentation of small objects.Inspired by feedback learning,this method proposes a dual feedback learning strategy,which adjusts the learning frequency of different samples based on the current learning of the model,and optimizes the allocation of learning resources to improve the colorization of small object categories.The experimental results demonstrate that this method can significantly improve the semantic consistency and appearance plausibility of small object categories during colorization.This dissertation presents a systematic investigation of the main challenges in the task of nighttime thermal infrared image colorization for traffic scenes.Inspired by important cognitive mechanisms such as visual attention,association,memory,and feedback,several high-performance colorization methods are proposed.The proposed methods are extensively evaluated on typical datasets,demonstrating significant improvements in semantic consistency,structure preservation,and appearance enhancement of small objects and small sample categories.These advancements are expected to provide important technical support for the development of all-weather multimodal visual perception techniques.Furthermore,this dissertation not only provides feasible research solutions for the implementation of weakly supervised thermal infrared image understanding,but also offers potential research insights for the design of brain-inspired visual computing models.