Research On Image Super-Resolution Via Deep Ensemble Learning

Given a low-resolution(LR)image,single image super-resolution(SR)aims at obtaining a high-resolution(HR)image with sharp details.As one of the fundamental techniques in the image processing area,image SR receives extensive attention from researchers,and finds wide applications in many real scenarios.Recent years,applying deep learning to the image SR task becomes a popular research topic.Based on the ensemble learning technique,this dissertation focuses on the research of the challenges unsolved in the deep learning based SR algorithms.The major contributions of this dissertation are summarized as follows.Firstly,an end-to-end image SR method via deep and shallow ensemble networks is proposed,which is to effectively increase the network depth and address the problem of performance saturation for deep SR models.While with strong learning capacity,deep networks can be easily to get stuck at sub-optimal solutions due to the difficulty of optimization.Therefore,this dissertation proposes to jointly train an ensemble of deep and shallow networks.The shallow network is easier to optimize and can restore the main structure of the image content,which significantly lowers the learning difficulty of the deep network and facilitates the gradient propagation during optimization.As a consequence,the deep network with stronger representation reconstructs fine details of HR images under the cooperation of the shallow network.To further improve the performance of the proposed method,the high frequency details are reconstructed in a multi-scale manner to simultaneously incorporate both short-and long-range contextual information.Besides,a learning approach is adopted to upsample the spatial size of images in the feature space.Experimental evaluations suggest the effectiveness of the proposed ensemble method.Secondly,an information-compensated image SR algorithm,which aims at enlarging the receptive field of SR networks and improving the context modeling ability of SR models.To this end,a novel network block,namely information-compensated downsampling(ICD)block,is proposed.To the best of our knowledge,the proposed ICD block is the first to apply pooling layers and pixel-wise LSTM to image SR task.It can not only effectively enlarge the receptive field of networks,but also compensate for information loss caused by the pooling operator.To capture the image context in a more comprehensive way and facilitate the training process,the common adopted pixel-wise LSTM with single direction is expanded to four directions and applied to the features in the coarsest resolutions.The final network consists of a stack of ICD blocks with dense connection and can be trained jointly.Compared to the commonly adopted approach that increases the network depth,the proposed ICD block and ensemble method provide more favorable solutions to enlarging the receptive field of network and achieving accurate HR reconstruction.Thirdly,a resolution-aware image SR algorithm is proposed,the goal of which is to obtain a single SR model that can deal with multiple upscaling factors with good generalization ability.It is observed that SR of multiple factors are essentially different but also share common operations.Based on this observation,we design an upsampling network(UNet)consisting of several sub-modules,where each sub-module implements an intermediate step of the overall image SR and can be shared by SR of different factors.A decision network(DNet)is further adopted to identify the quality of the input LR image.The two networks together constitute the proposed model,which can adaptively select suitable sub-modules of UNet to perform SR upon the decision of DNet and simultaneously perform SR with multiple upscaling factors.A new hierarchical and multi-task loss function is further proposed to jointly train the ensemble model on SR tasks of multiple factors.Experimental evaluations demonstrate that the proposed method compares favorably against state-of-the-art methods and better generalizes across different up-scaling factors.Finally,a blind image SR method via a mixture model is proposed to handle the unknown blur kernels during the SR process.This algorithm clusters SR tasks of different blur kernels into a set of groups.Each group is composed of correlated SR tasks with similar blur kernels and can be effectively handled by a combination of specific networks in the mixture model.An encoder network is designed to model the blur kernel with a latent variable.Based on the latent variable,this mixture model achieves automatic SR tasks clustering and network selection.The proposed method is built upon the maximum likelihood.To jointly train the whole model,a lower bound of the likelihood function is derived,which circumvents the intractability in direct maximum likelihood estimation.Besides,a pre-training strategy is proposed for the encoder network to avoid trivial solutions during joint training.Extensive evaluations justify that the effectiveness and generalization of the proposed image blind SR method.