Face Super-resolution Via Consistent Manifold Learning

In the detection of the criminal investigation, the face image of the suspect is the most concerned goals for investigators. But in the actual monitoring, the suspect is far from the camera and the resolution of the surveillance camera is so limited, which lead to the size of the captured face image is small. Thus, it is difficult to identify the suspect. How to improve the spatial resolution and quality of the captured face, and enhance image clarity and discriminability of face value is a challenging field. Face super-resolution is the technology aiming at solving the problem of inducing one high-quality and high-resolution (HR) face image from the given low-quality and low-resolution (LR) face images by combining with complementary information from consequence image frames or learning the prior knowledge from the given image database.In recent years, a number of learning based face image super-resolution approaches has been proposed. Based on the assumption that the LR and HR manifold spaces share the similar local structure, these learning based face image super-resolution approaches try to learn the relationship between the LR and HR training sets, and then use the learned model to infer the HR face image of the input LR image. But face image super-resolution technology still faces the following three challenges especially in the surveillance conditions:(ⅰ) the high-dimensional space consisted by the high-dimensional data is not linear. The size of current face database is limited;(ⅱ) the manifold space is susceptible to noise. In serious case, the face image of suspect often severely affected by noise;(ⅲ) the local structure of the low and high resolution image manifold spaces is inconsistent. In this thesis, we study the problem of enhancing the representation ability of the given database, designing a robust image patch representation, learning the neighbor embedding model with consistent coupled manifolds, and the main contribution of the paper is highlighted in the following:(1) Face database representation expansion through nearest feature line manifold learning.To solve the inconsistence between the LR and HR manifold spaces caused by the limited sample in the face database, we introduce the concept of nearest feature line, and it can extend every two sample points of the training set to numberless points on the feature line by connecting the two points, which improves the expression ability of the original samples. Our algorithm maintains linear relationship in a smaller local space than traditional manifold learning based methods, and reveals the non-linear relationship between HR and LR face manifold space. This reflects the nature of manifold learning theory. Experimental results show that the gain of the proposed face super-resolution algorithm over [3] is1dB in term of PSNR in the CAS-PEAL-R1face database [4].(2) Noise robust patch representation via Locality-Constrained Representation.The noise will reduce the accuracy and stability of sparse representation based face super-resolution method. To this end, we propose a noise robust face super-resolution method via locality-constrained representation. It analysis the influence of noise on the representation of image patch, and establish a locality-constrained representation model, which uses the locality prior to constraint the representation. This method can project each image patch into its neighborhoods in the training set adaptively. leading to a robust inage patch representation, which solves the not unique problem of least square representation [1] and the unstable problem of sparse representation [2]. The average PSNR and SSIM improvements of locality-constrained representation method over the sparse representation method [2] are1.02dB and0.0168respectively in the FEI face database [5]. The subjective results of reconstructed face images are also better.(3) Projection between the LR and HR spaces through HR manifold constraint.The current manifold learning based super-resolution approaches mainly focus on the structure of the LR manifold and neglect the HR one. To this end, we propose to establish a HR manifold constrained mapping model between the LR and HR images. In particular, we use the original HR manifold structure to regularize the reconstructed HR manifold space, making the local geometric structure of the reconstructed HR image patch manifold and the original HR image patch manifold in consistency. When compared with the LR manifold constrained face Super-resolution method, the gain of the proposed method is0.34dB and0.0044in terms of PSNR and SSIM in CAS-PEAL-R1face database [4],0.27dB and0.0049in terms of PSNR and SSIM in AR face database m.(4) Learning and fitting of the image degeneration model via Multilayer Locality-Constrained Ite rative Neighbor Embedding and Intermediate Dictionary Learning.The performance of learning based face super-resolution methods will drop sharply when the degeneration process is very complex. To this end, we introduce the LR dictionary learning to the face image super-resolution task for the first time, and establish a multilayer locality-constrained iterative neighbor embedding model and an intermediate dictionary learning model. Then we propose to update the LR training set by constructing intermediate dictionaries and modeling the relationship in much more consistent LR and HR spaces. Experimental results show that the gain of the proposed face super-resolution algorithm over [2] is1.66dB and0.0303in terms of PSNR and SSIM in the CAS-PEAL-R1face database [4].In summary, this thesis completes the express expansion of face image database, robust face image representation and learning of the relationship between high and low resolution images by referring the mechanism of human visual perception and cognition. It provides a new way to the actual monitor face image super-resolution reconstruction on the theory and key technologies.