Glass Detection And Mirror Detection Based On Deep Learning

Glass detection aims to recognize the glass surface(s)in an image and segment the target regions.Similar to glass detection,mirror detection aims to detect and segment the mirror surface(s)in an image.However,the appearances of glass surfaces and mirror surfaces mainly depend on their surroundings,thus it is difficult for existing computer vision systems to detect their presence and affects many vision tasks such as depth estimation and instance segmentation,as well as the proper functioning of autonomous systems such as robots,autonomous driving and drones.In order to detect glass surfaces and mirror surfaces in daily scenes,we propose a deep-learning based mirror detection method and a deep-learning based glass detection method.On the one hand,we propose a novel approach for glass surface detection with ghosting cues in RGB images.We observe that ghosting effects typically appear on glass surfaces,as each piece of glass has two contact surfaces causing two slightly offset layers of reflections.In this paper,we propose to take advantage of this intrinsic property of glass surfaces and apply it to glass surface detection,with two main technical novelties.First,we formulate a ghosting image formation model to describe the intensity and spatial relations among the main reflections and the background transmission within the glass region.Based on this model,we construct a new Glass Surface Ghosting Dataset(GSGD),which contains ～3.5K samples,to facilitate the glass surface detection process.Second,we propose a novel network for glass surface detection.Our method consists of a Ghosting Effects Detection(GED)module and a Glass Surface Detection(GSD)module.The key component of our GED module is a novel Double Reflection Estimation(DRE)block that models the spatial offsets of reflection layers for ghosting effect detection.The detected ghosting effects are then used to guide the GSD module for glass surface detection.Extensive experiments on our proposed GSGD and two benchmark datasets demonstrate that our method outperforms the state-of-the-art methods.On the other hand,we propose a LF-based mirror detection network.We propose a threestream cascaded network to detect mirror surfaces by integrating multiple light field visual cues.First,we propose a novel representation of light field called macro-pixel image stack(Mac PIStack),which consists of several macro-pixel images composed of different sub-view images,to make full use of parallax information and geometric information of light field for mirror detection.Then,we design a cross-modal fusion module based on comparison and compensation strategy to aggregate multi-modal backbone features.Finally,we propose an attention module to aggregate contour information for mirror detection.For network training and validation,we propose a light field mirror detection dataset with 1520 samples.Extensive experiments demonstrate that our proposed model outperforms state-of-the-art methods from relevant fields.