Multi-modal Image Matching Based On RGB-D-T Information

RGB-D-T multi-modal image matching technology is a modern high-tech that combines sensor measurement,digital image processing,signal processing and other technologies.RGBD-T multi-modal image matching technology can achieve pixel-level matching between multi modal images and can comprehensively utilize the complementary advantages between multi modal images,so it can be applied to environment perception,scene understanding and other fields.In this paper,the RGB-D-T multi-modal imaging system is built with KinectV2 camera and FLIRA615 thermal infrared camera.The KinectV2 camera includes a color camera and a depth camera.The key technologies such as calibration of RGB-D-T multi-modal imaging system,multi-modal image matching,hole filling of three-dimensional color field and threedimensional temperature field have been thoroughly explored and researched.The main research work is as follows:(1)In the calibration of RGB-D-T multi-modal imaging system,Zhang Zhengyou calibration method is used to complete the calibration of color camera and depth camera.Aiming at the problem that thermal infrared camera can not be calibrated by ordinary checkerboard calibration board,a thermal infrared calibration board is designed and manufactured,and a calibration point detection algorithm based on the geometrical shape of thermal infrared calibration board is proposed.The experimental results show that the accuracy of thermal infrared camera calibration can be improved by using the calibration point detection algorithm based on the geometrical shape of thermal infrared calibration board.(2)In the aspect of multi-modal image matching,aiming at the problem that traditional image feature point detection methods can not be used to complete pixel-level matching among heterogeneous images.Thus,the projection transformation matching algorithm is adopted based on the parameter matrix of each camera in RGB-D-T multi-modal imaging system and the external parameter matrix representing the spatial position relationship between them.The projection transformation matching algorithm completes the pixel-level matching of RGB-D-T multi-modal images,the RGB-D-T information field is constructed based on the matching results,and the RGB-D-T information field is visualized by reconstructing three-dimensional color field and three-dimensional temperature field with point cloud matching relationship.The experimental results show that the projection transformation matching algorithm can achieve accurate pixel-level matching of multi-modal images.(3)In the aspect of filling holes in three-dimensional color field and three-dimensional temperature field,through the analysis,it is found that the depth image in RGB-D-T multimodal image has noise points,and the matching between the depth image pixels and the color image pixels and the thermal infrared image pixels can not be completed naturally at the noise points.Filling holes in the three-dimensional color field and the three-dimensional temperature field needs to be accomplished by depth image denoising.In this paper,the convolutional Autoencoder is designed to complete the depth image denoising work.The experimental results show that the convolutional self-encoder designed in this paper has excellent denoising performance and improves the quality of reconstructed three-dimensional color field and threedimensional temperature field.