Image Super-resolution Based On Deep Reinforcement Learning

Image super-resolution(SR),a process to reconstruct a high-resolution(HR)image from a low-resolution(LR)one is a challenging task.Due to its ill-posed nature,it is an irreversible problem.The main point of the task is how to estimate unknown pixels of HR image by known pixels of LR image.In recent years,due to the development of deep learning,a large number of researchers have focused on this field,and many excellent algorithms have emerged.One the one hand,the algorithms based on deep learning are all learning global texture mapping.However,one easily notes that any SR algorithm/network might not perform optimally on all types of image regions(i.e.,associated with different textures or semantics).Hence,using the same network structure for global image reconstruction can not get the optimal SR image.On the other hand,due to the breakthrough of reinforcement learning,more and more researchers have applied reinforcement learning in the computer vision.This motivates us to propose a novel framework which dynamically selects the best SR model for each specific image region in a collaborative manner,given a pool of off-the-shelf SR algorithm or pre-trained baseline SR networks.En-capsuled in a deep reinforcement learning framework,for each pre-segmented image region,a policy selector is proposed to dynamically select a proper algorithm/baseline SR network from the SR algorithm pool.The contributions of this paper are as follows.Firstly,this paper proposes an dynamic support selection framework for image SR.Based on deep Q learning algorithm,the STATE is defined as features of image regions,which is extracted by three-depth convolutional neural network.While,the ACTION is the SR model selected by the framework.Then,according to our proposed Within-region reward,the dynamic strategy selection model of the image region is obtained.Secondly,this paper proposes a new reward function named Between-Region Quality Reward.it considers one cases: adjacent image regions favor similar SR models to apply on them,as dissimilar SR models might present different algorithmic behaviors,which results in artifacts at boundaries between regions.We combined these two reward function into a joint reward function.According to the joint reward function,we improved the DQN framework.Hence,the artifacts problem is solved.Thirdly,in order to prove the advantages of deep reinforcement learning selection,an image super-resolution algorithm based on model fusion is proposed.We compare this method with our proposed dynamic strategy selection framework based on deep reinforcement learning.The experimental results prove the advantages of our dynamic strategy selection.Finally,Extensive experimental studies on several SR benchmarks demonstrate that the proposed scheme effectively pursuits a good collaboration among all SR algorithms in the pool and outperforms prior art.And parameter sensitivity study and computational complexity comparison are also performed.To the best of knowledge,this work is the first attempt to employ reinforcement learning for model fusion in image super-resolution task,and it is a general fusion framework which can plug-in any types of off-the-shelf SR algorithms.