Research On Pixel Design For Multisampled Imaging

Multisampled imaging is using a single sensor to simultaneously sample multiple imaging dimensions(e.g.,space,time,spectrum,and brightness),which depends on the application.It includes the color imaging with a color filter array(CFA),high dynam-ic range(HDR)imaging using the spatially varying exposures(SVE)technology,and multi-image super-resolution.Over the past decades,a great development on this topic has been achieved.However,there are still some issues that need to be considered.First,most of the existing sensors for multisampled imaging use the square pixels in regular layouts.However,previous study has shown the advantages of the irregular layout of an-imal retina in producing high-quality images.So the effectiveness of irregular layouts for multisampled imaging is worth studying.Second,many pixel design approaches for mul-tisampled imaging overlook the characteristics of subsequent reconstruction algorithms.In order to enhance the performance of a given type of reconstruction algorithms,one should design pixels specifically for it,where the reconstruction process is fully consid-ered in the design process.In this dissertation,we focus on the above crucial problems,and our main contribu-tions include:(1)We propose the Penrose pixel layout for demosaicking and present the Ammann-Beenker pixel layout for both super-resolution and demosaicking.The two pixel layouts are aperiodic,uniformly three-colorable,and use only two simple pixel shapes.These properties make them advantageous over other irregular layouts in being used as imaging sensors.We evaluate the performance of the two irregu-lar pixel layouts on both super-resolution and demosaicking.Experimental results show that Penrose and Ammann-Beenker pixel layouts outperform the square one.(2)We present the Penrose pixel layout for HDR imaging using the SVE technology,making it aperiodic in both exposure and pixel arrangement.Since the Penrose pixel layout is irregular and aperiodic,the existing HDR reconstruction methods are not applicable to it.So we develop a new HDR reconstruction method with a Gaussian mixture model(GMM)model for regularization.Extensive experiments show that Penrose pixel layout is advantageous in alleviating the reduction in spatial resolution of the reconstructed HDR images.(3)Based on the frequency structure,we develop a new method to automatically design CFAs specifically for frequency selection based demosaicking.Then we extend this approach to design high-sensitivity CFAs with panchromatic pixels in the frequen-cy domain,which has a mathematical model and is fully automatic.To accomplish this,we formulate high-sensitivity CFA design with panchromatic pixels as a multi-objective optimization problem,which simultaneously maximizes the robustness to aliasing artifacts and the percentage of panchromatic pixels.Extensive experiments on both low-light and normal-light datasets demonstrate the superiority of our de-sign method.(4)We present a theoretically grounded approach to design specific CFAs for sparse representation based demosaicking,where the dictionary is well-chosen and fixed.We formulate the CFA design as the minimization of the mutual coherence with the CFA's physical realizability constraints,where most methods for minimizing the mutual coherence do not apply.We develop a new method to solve it based on generalized fractional programming.Extensive experiments on benchmark images demonstrate the superiority of our design method.