A Study Of Super-resolution Methods From Natural Image To Multiphoton Medical Imaging

High-resolution(HR)images often preserve more details and critical information that play a crucial role in numerous fields,such as surveillance,medical diagnosis,life science research and remote monitoring for crops.However,it is difficult to obtain an HR image in practical applications due to the diffraction limit of optical devices,the disturbance of the noise existing in the imaging system and the downsampling operation during the optical-electrical conversion.The goal of super-resolution(SR)is to recover the original HR image from one or more low-resolution(LR)images based upon reasonable assumptions or prior knowledge about the imaging process.Enhancing images by SR technique,it is an effective technique to improve some computer vision tasks,for instance boosting the accuracy of face recognition,improving the precision of semantic segmentation and enhancing the accuracy of computer-aided diagnosis system(CAD).SR has received lots of attention in the computer vision community.Multiphoton microscopy(MPM),which breaks the diffraction limit inherent in the traditional optical system,is an advanced medical imaging based upon two-photon excitation fluorescence(TPEF)signal and second-harmonic generation(SHG)signal that are excited during the interaction between laser and biological tissue.It has been extensively used in various fields of life science and biomedical science.Nevertheless,MPM imaging is susceptible to the dispersion and the absorption of the light in the samples,and the complicated optical system.So it is important to study SR approaches for enhancing MPM images.Resulting from the development of GPU and big data,deep learning(DL)has achieved great success in many computer vision tasks and attracted much attention.A deteriorated model for MPM imaging is proposed in the dissertation via the analysis on the principle of MPM imaging.By comparing the deteriorated models between natural images and MPM images,there are some homogeneous nonlinear mapping relationships between them.So it is better to research the SR and enhanced methods for MPM images via exploiting the DL techniques beginning from natural images with both supervised and unsupervised learning.The main contributions include:1.We propose a new image SR method,which combines the upsampling and deblurring process into a deep convolutional network.To address the downsampling and blurred problems occurred in the deteriorated process,a method combining the reconstruction and deblurring operation is proposed,which can be trained by end-to-end.The function of each layer in the network during the image reconstruction pipeline is first studied.After understanding the principle of convolutional network,the first layer of the network is initialized with different parameters of Gabor filters and the others are randomly initialized with Gaussian distribution.In addition,the quality of the reconstructed image is boosted by virtue of combining the reconstruction and deblurring into a unified framework.It is worth noting that the proposed approach is more robust to blurred images despite no blurred elements in the train set.The related result has been published in the International Conference on Intelligent Computing.Springer,Cham,2017: 338-344.(EI)2.Based upon dilated convolutional neural network,we propose a new image SR method which is able to quickly enlarge the field of view(FOV)of network and support multiple scales.The quality of the reconstructed image is related to the size of FOV exploited during the recovering process.For that an efficient method to enlarge the size of FOV is proposed.Meanwhile,a multi-magnification network for image SR is designed.By reasonable assigning the dilated rate in each layer,the proposed method achieves notable improvement in terms of both quantitative and qualitative measurements.One special upscaling factor commonly requires one network trained from scratch,which is time-consuming.Therefore,a cascaded convolutional network for supporting multiple scaling factors is developed.The experimental results demonstrate that the cascaded model can further boost the SR performance.The related result has been published in Journal of Neurocomputing,2018,275: 1219-1230.(SCI-?,IF: 3.317)3.Based upon generative adversarial nets,we propose an unsupervised-learning method for image SR.It requires pairs of LR-HR images for most of the example-based SR methods during the training stage.It is a challenge for training example-based methods without pairs of LR-HR images.We propose a deep unsupervised-learning approach for image SR with generative adversarial network.It doesn't require pairs of LR-HR images for training our method.The proposed method utilizes the adversarial framework to learn the natural image manifold from the original HR natural images,which will help the generator network to generate photo-realistic images.There only exist deteriorated LR images for training the generator network.To keep the consistency in geometry between the input LR images and the synthetic HR images,several downsampling methods are compared according to the image deterioration.Additionally,a mixed regularizer is proposed to keep the smoothness of image continual section and to preserve the sharp edge in the produced images.Comparisons with several state-of-the-art supervised learning-based methods,experimental results show that the proposed approach achieves a comparable result in terms of both quantitative and qualitative measurements,and it also implies the feasibility and effectiveness of the proposed unsupervised learning-based single-image super-resolution algorithm.The related result has been published in Journal of Signal Processing: Image Communication,2018,68: 88-100.(SCI-?,IF: 2.244)4.Based upon the residual module,we propose a residual-learning MPM image SR method.The quality of MPM images is sensitive to the inner tissue of the samples and the complex optical system.After the study on the natural image SR methods and the deteriorated models between natural image and MPM image,a SR method for MPM image is proposed according to the Retinex theory.The proposed method combines the reconstruction operation and enhancement operation into one framework,which can be trained by end-to-end.To evaluate the effectiveness of natural image SR method for MPM image,two aforementioned SR methods are trained from scratch with MPM images.The results indicate that natural image SR methods can be transferred to MPM image SR.Due to the more complicated deteriorated process in MPM imaging than in natural image,the aforementioned SR methods are slight deficient in nonlinear mapping because of exploiting a few convolutional layers in the models.As a consequence,on the basis of the characteristics of MPM image we employ the residual model to develop a deeper network,which has a more powerful nonlinear mapping.We also compared the proposed method with EDSR,which was the winner of NTIRE 2017 challenges.The experimental results show that our method is more effective for MPM image than EDSR.Some of the related results have been published in Neurophotonics,6(4),045008(2019)(SCI-?,IF: 3.581)and Journal of Biophotonics(SCI-?,IF: 3.763).