Research On Key Technologies Of Digital Film Restoration

Digital film restoration is a technique for detecting and removing various kinds of defects in digitalized films using image and video processing approaches, which is considered as a successful application of computer techniques in film restoration field. It converts seriously damaged old films into high definition videos with reduced artifacts that meet the visual preferences of the audiences. This will not only revival the commercial values of old films, but also proliferate program reservoir for the Television broadcasters. From the scientific point of view, the studies on digital film restoration prompts the developments of many other research areas, such as artificial intelligence, computer vision and image understanding. This dissertation first introduces some background knowledges and system level workflows of digital film restoration algorithms, followed by some theoretical reviews on mathematical tools used in the dissertation such as Overcomplete Wavelet Expansion (OWE). The dissertation then discusses the detection and removal of three of the most prominent defects in old films, namely line scratch, blotch and grain noise. For the treatments of those artifacts, specific mathematical models are introduced, analyzed and compared, based on which several improved artifacts processing methods are proposed and verified under practical scenarios. The detailed layout of this dissertation is as follows:In Chapter one, we first review the histories of films and film industry, together with some introduction to current situations of films' preservation. After that, we discuss the limitations of manual film reparation techniques which underline the significance of computer based digital film restoration methods. And then we go through some key issues as well as systematic workflows in digital film restoration, with emphasis on the interactions among different system modules. In chapter two, we describe the theoretical basis of the proposed film restoration algorithms. We begin with a survey of wavelet as a multi-resolution image analysis tool. Particularly, we use OWE as the mathematical basis of the algorithms developed in this dissertation. The OWE based film restoration algorithms are then introduced in detail. We then review the theories of MRF and Simulated Annealing (SA) as one of its realization approach, which are frequently used in line scratch and blotch processing. The Expectation Maximization (EM) method is recapped at the end of this chapter, which is used in grain noise removal and spatial verification of blotch detection.In chapter three, the algorithms for detecting and removing line scratches are introduced in detail. This chapter begins with the review of perceptual properties of scratch artifacts as well as several classic detection and removal methods. Much emphasis is put on the mathematical modeling of line scratches, which can be further classified into primary scratches, secondary scratches and scattered scratches, according to differences in their geometrical shapes and statistical distributions. The OSA algorithm for detecting and suppressing primary and secondary line scratches is firstly introduced. This algorithm works in the OWE domain and is more accurate than spatial domain methods in that it provides 2D location of the scratches. This is because the horizontal wavelet coefficients are more sensitive to the vertical line scratches. And through multiple-constraining and morphological optimization, the algorithm can further categorize the artifacts into primary and secondary scratches. During the restoration stage, the low-pass bands coefficients are completed through linear interpolated while the high-pass bands are repaired using 1D interpolation with LMMSE principle. This scheme is capable to recover more natural textural information and exempts the oversmoothing effect as well as interpolation errors associated with many conventional methods. The OMEA algorithm is proposed to deal with scattered line scratches that have generally been overlooked by conventional methods. The scattered scratches cannot be detected with vertical projection based method due to the artifacts'variations in gray-level and length and their random appearance in the whole frame. In the OMEA algorithm, the scattered scratches are firstly enhanced through pattern matching in the horizontal subbands of the OWE coefficients. And then MRF is introduced to model the constraints between the artifacts and neighboring areas. Those scratches are then processed with improved exemplar-based inpainting technique that restores both image textures and structures. Empirical results show that the proposed OSA and OMEA algorithms can detect and restore line scratches more effectively than conventional methods.In Chapter four, the problems of blotch detection and removal are studied. The classic algorithms are reviewed before introducing a novel multi-step blotch detection and verification algorithm named MDV. In MDV's preliminary step, the rough locations of blotches are found through SDIp and SROD algorithms. And then those candidate locations are refined with an improved MRF-based method. The improved MRF method is more robust to noises than conventional methods because the original-frame information and a denoising factor are added into the model. Furthermore, the total computational complexity is largely reduced over classical MRF-based methods because of the preliminary detection stage. The detection results are then verified with a spatiotemporal matching and a spatial domain EM algorithm. The blotches restoration algorithm 3D IEI is then introduced in detail. 3D IEI is a novel 3D spatiotemporal exemplar-based inpainting algorithm. As compared with conventional methods, 3D IEI can restore much more textures in the frame and prevents oversmoothing. Also, by applying a spatial constraint over matching regions in consecutive frames, the computation cost is largely saved in 3D IEI. Based on the experimental results over several image sequences containing various kinds of blotch, the proposed MDV and 3D IEI algorithms are proved to be able to detect and remove the blotch in a faster and better way than traditional methods.Finally, Chapter five deals with grain noise. The properties, causes, structures, mathematical models of grain noises, as well as some representative denoising algorithms are firstly reviewed in detail. A denoising algorithm HSS is then proposed. In HSS algorithm, the noise components of the image frames are firstly extracted using OWE. The variance of the noise is then estimated using a two-step multiple-region EM method. This algorithm adaptively estimates variances for different regions and reduces the influence of edges and interactions between image regions. The detected noise variances are then classified into obvious- and mix- types according to their visual saliency features to the human visual system. The two types of noises are further processed with LMMSE filter and NLM filter. The NLM filtering stage uses the global segmentation template information in determining similarities between image blocks. As compared to classical NLM filter, this approach reduces the similarity level between image blocks from different regions and prevents their mutual interference during the restoration of image contents. This helps to preserve more image details as well as enhance the denoising performance of the algorithm. Empirical results show that in the denoising process over the images added by two kinds of grain noise, the propose HSS algorithm can remove the noise more effectively with better preservation of edges and textures than conventional methods, and this superiority is verified in visual perception and two image quality assessment standards.