Specular Highlight Detection And Removal In Natural Scenes

Specular highlight as a common physical phenomenon in the real world usually appears as bright spot on shiny object when illuminated.Most natural images in daily life contain specular highlights.Specular highlight detection and removal are important tasks in computer vision and computer graphics,and significantly benefit other tasks,such as light source position estimation,segmentation,recognition,recoloring,object shape estimation,and illumination estimation.Moreover,they are broadly used in daily life and production,such as automobile surface check,industrial robot automatic assembly,automatic driving,and smartphone photography.Therefore,it is very important to perform specular highlight detection and removal in the process of image understanding and editing.Although researchers have proposed many specular highlight detection and removal methods,there are still quite a few problems that remain unsolved,especially for natural images often due to achromatic surfaces,white noise,heavy texture,chromatic illumination,and so on.Existing specular highlight detection methods are mostly based on thresholding.Hence these methods do not have the ability of semantic perception and high-level prior knowledge,resulting in the misjudgment of some achromatic regions(such as white material surfaces,texts,and visible light sources)in the background as specular highlights.Moreover,existing specular highlight removal methods are mostly based on traditional optimization.Hence these methods are less able to solve the wellknown hue-saturation ambiguity well and fail to restore the color and texture details under specular highlights,resulting in removal results with color distortion and unreal feeling.In addition,there is no large-scale real publicly available dataset for specular highlight detection and/or removal,which seriously restricts the in-depth study of specular highlight detection and removal today when deep learning tools are widely used.To address the above problems,this thesis presents a set of effective solutions,including the presented large-scale real specular highlight detection and removal datasets,and the proposed specular highlight detection and removal methods.From the perspective of the traditional prior knowledge,this thesis firstly studies the specular highlight removal problem based on the dichromatic reflection model.Next,from the perspective of deep learning,this thesis explores the specular highlight detection problem.Finally,this thesis studies how to combine the specular highlight detection task and the specular highlight removal task into a unified network to thus achieve the mutual gain of the two tasks.Specifically,this thesis mainly includes the following three contributions:(1)This thesis proposes a novel specular highlight removal method based on priors of the natural scene,which can effectively remove specular highlights while avoiding visual noticeable artifacts such as black color block and color distortion in the background.This thesis observes that specular highlights in natural scenes have two characteristics.First,the specular highlight is often small in size and sparse in distribution.Second,the remaining diffuse component can be represented by a linear combination of a small number of basis colors with sparse encoding coefficients.Based on the two observations,this thesis designs an optimization framework for specular highlight removal based on two sparse constraints.The proposed method can remove specular highlights of those achromatic pixels while preserving the saturation of the background,resulting in natural-looking results.(2)This thesis presents the first large-scale natural image dataset for specular highlight detection(named SRW),which fills the gap in the lack of large-scale real dataset in the specular highlight detection field;This thesis proposes a deep specular highlight detection network based on context contrasted features,which can effectively overcome the ambiguity between specular highlight and background.The SRW dataset covers the most common scenes and materials in daily life on which specular highlights easily appear,and contains 4310 images in total,each with a labeled ground truth mask.The SRW dataset is the largest publicly available dataset in the world.The proposed method can accurately locate specular highlights of varying sizes,while effectively avoiding obstruction from the background.(3)This thesis presents the first large-scale natural image dataset for both specular highlight detection and removal(named SHIQ),which fills the gap in the lack of real dataset in the specular highlight removal field;This thesis proposes a multi-task network for joint specular highlight detection and removal based on information propagation across rows and columns,which can effectively overcome the huesaturation ambiguity caused by existing methods.The SHIQ dataset contains 16 K images in total,each image with corresponding specular highlight detection and removal ground truths.The SHIQ dataset is the largest publicly available dataset in the world,which can simultaneously for specular highlight detection and removal.The proposed network can effectively remove specular highlights while recovering the color and texture details under them,thus yielding high-quality results without visually noticeable artifacts.Focusing on specular highlight detection and removal,this thesis deeply and systematically studies them,and basically solve several key technical problems in the field.The experimental results show that the proposed methods can more effectively overcome the highlight-background ambiguity problem and the hue-saturation ambiguity problem,thus yielding high-quality detection and removal results.