Feature Learning And Patch Matching Of Multispectral Images Based On Deep Neural Networks

Image matching based on patches has always been the important basis in the field of computer vision.Find the precise matching relationship between image patches plays a vital role in many image task.Multispectral images with different imaging mechanism have grey information of nonlinear relation which results in that image patch matching method based on traditional image descriptors in multispectral images is hard to achieve better effect.Aiming at the problem that multispectral images have difficulty in accurate matching and considering of the powerful feature learning ability of deep neural network,this paper proposed a patch match method based on convolutional neural networks to realize more accurate patch matching between infrared and visible light image patches.This paper first introduced the infrared-visible image patches dataset which is pictured and built by ourselves.In task of deep learning,a well dataset is relate to the generalization ability and robustness of final trained models.We collected images of the same scene with an infrared camera,a visible light camera and a filming stand.And we use MATLAB to realize the image processing algorithm which includes the affine transformation of original images,patch cropping with sliding windows and the generation of positive and negative samples.We have divided our dataset into three parts which are the training,validation and testing dataset.After that,this paper introduces the realization of four kind of patch-match Convolutional Neural Networks,includeing three pre-existing networks,the 2ch net,the 2ch-deep net,the MatchNet and a CNN model that we newly proposed which is called the residual siamese network.The residual Siamese network is an innovative network that takes the advantages of the residual network and the MatchNet.Through the comparison between our proposed method and pre-existing models,it can be shown that the residual siamese network can perform better in patch matching task of infrared-visible image pairs.Besides that,in considering of the parallax between image pairs,we come up with the ideal of combining the spatial transformation network into one of two branches of our residual siamese network to increase the space transform invariance of our method.Finally,we make a testing dataset of image pairs with planty of artificially spatial transformation features to test our ideal.The result shows that the joining of STN in the residual siamese network provides much more powerful generalization ability of parallactic image pairs.