A Study On Super-resolution Of Multi-source Images

Images are always in low-resolution with blurred details when they are captured by the low-end cameras or received from the bandwidth limited internet environment. These degraded images will affect the visual experience and impose a burden for further computer vision applications. Therefore, super-resolution has been studied for decades as it is a fundamental problem in the field of image processing.Image super-resolution refers a digital image processing technique that reconstructs a high resolution image from a single or several low resolution observed images. Based on the catagories of images that need to be super-resolved, image super-resolution includes of color image super-resolution and depth image super-resolution. With the development of the research on this field, super-resoultion for co-occurred color image and depth image has been proposed and becomes a hot topic in recent years. Under this circumstance, we focus on studying the super-resolution method for multi-source images, i.e., co-occurred color image and depth image. In this thesis, several super-resolution methods have been proposed and can serve as feasible solutions or valuable supports for some image super-resolution problems.Research on the super-resolution for co-occurred color image and depth image should include discussions about the acquisition of depth information by using different measurement technques, which exhibits differnet properties. In this thesis, we focus our study on two kinds of3D vision systems that can provide depth information, which are multi-view vision system and RGBD vision system. Based on some image priors and new models, we proposed several novel super-reslution methods.We start our research on the color image super-resolution problem in multi-view stereo vision system. Recently, the mixed-resolution coding/decoding approach for multi-view images/videos, in which low-resolution and full-resolution images are jointly used, has been proposed to reduce the amount of data. It serves as a feasible transmission solution for bandwidth limited internet environment. But the asymmetric resolutions between different views will limit their further applications. Therefore, several super-resolution techniques for mixed-resolution multi-view images/videos have been proposed. In this thesis, we propose a new depth based super-resolution algorithm for mixed-resolution multi-view images, under the guidance of the depth fusion result estimated from different views. First, we match the high resolution image of the reference view with the initial super-resolved images of low resolution views to get the initial depth maps, and apply median filtering on them to obtain the fusion result. Then, under the guidance of the depth information, we proposed a super-resolution algorithm by mixing the disparity-based mapping result and nonlocal reconstruction result together in a fusion manner. Finally, we use the super-resolved results to update the low resolution view and repeat the above two steps until obtaining the stable depth estimation and super-resolution results. To evaluate the effectiveness of the peropsoed algorithm, we conducted several experiments on Middlebury dataset and compared it with other methods. The results show that the proposed method achieved better results than others both in objective indices and subjective visual experience.We continue our research on the super-resolution problem in asymmetric stereo video. After the analysis on the influence factors for depth estimation and super-resolution in it, we point out that they are indeed two coupled problems. Based on this observation, we propose a novel method that can modeling these two problems in a unified scheme. First, we model the depth estimation problem by constructing an energy function which consists of a data term, a smoothness term, an occlusion term and a temporal consistency term. Then, under the guidance of the depth information, we apply the imaging model constraint, view correspondence constraint and nonlocal constraint on the super-resolution problem and formulate it into a unified energy function after adding the above one. Finally, we use an alternating optimization technique to obtain the depth estimation and super-resolution results simultaneously. We conducted several detailed experiments on two public stereo video datasets and compared the proposed method with state-of-the-art methods both in objective indices and subjective3D visual experience. The experimental results convince that the proposed method has superiority in obtaining better depth estimation and super-resolution results simultaneously than other methods.Then, we move on to the depth map super-resolution topic in the RGBD vision system and propose two novel depth super-resolution methods by using the color information. First, we derive two nonlocal imaged guided filtering methods based on the nonlocal linear model which is inspired by the nonlocal similarity prior. After obtaining their explicit kernel functions, we discuss their differences and relations with image guided filter and nonlocal means. Experimental results on public datasets convince the effectiveness and superiority of the proposed filters for depth super-resolution, depth enhancement, image denoising and image dehazing. Later in our research, we propose a new depth super-resolution method via local and nonlocal prior aiming at a more robust technique than others for low quality initial depth map. First, we model the problem by constructing an energy function based on similarities between neighboring pixels in color domain and depth domain. Then, we proposed a fast approximation algorithm after the analysis on the optimization about the energy function. Experimental results on three public datasets convince the superiority of the proposed method in obtaining better super-resolution results. In addition, it is robust to low quality initial depth map.Finally, we propose a stereo-vision-assisted3D vision system that takes the advantages of stereo vision system and RGBD vision system, where we can capture two high resolution color images as well as one low resolution depth image simultaneously. Based on this system, we propose a novel depth super-resolution method by using the depth information from two sources. First, we obtain the initial depth super-resolution result by using the above technique we proposed. Then, we calculate the disparity map by matching these two high resolution color images, where we use an adaptive weighted local matching technique by measuring the similarities between neighboring pixels in color domain and depth domain. We then obtain the corresponding depth map based on the optical triangulation, which may contain lots of wrong pixels in occlusion regions and textureless regions. Finally, we use pixels from the above super-resolution result to replace these wrong pixels by using an adaptive weighted local filling technique. We conducted several experiments on Middlebury dataset to evaluate the performance of the proposed method. The results show that the proposed method can achieve better depth result than the related super-resolution methods as well as the stereo matching techniques.