Spatio-temporally Coherent Video Shadow Detection And Removal

Shadow,which is a common optical phenomenon in nature,usually has bad effect on many computer vision tasks such as object detection/recognition,image/video segmentation and intrinsic decomposition and etc.Video shadow removal is an important research topic in computer vison and graphics,which aims to detect and remove shadows in the video and recover the scene information in the shadow regions under normal illumination and keeps the spatio-temporal coherence of the video.Video shadow removal has significant application values on film post-processing,vision algorithms optimization,object tracking and recognition in video,video content editing and so on.Video shadow detection and removal is also a challenging research topic at the same time.Firstly,shadows exists in the video are usually complex and diverse,and there are probably moving cast shadows as well as static shadows with varying forms relative to the camera,which is challenging for shadow detection.Then,the camera may have complex movements consist of translation and rotation.Traditional background modeling-based methods usually can't handle video scenarios with free moving camera,and the free moving camera makes it hard to keep the spatio-temporal coherence of the video after shadow removal.Finally,the texture details on the shadow boundaries are lost seriously due to the sudden illumination change.It is also a challenging problem to make the shadow boundaries transit smoothly and naturally after shadow removal.Focusing on the above key problems,this paper conducts a relatively systematic research on shadow detection and removal for video.We firstly detect the accurate shadow regions in the video using a fast matting-based interactive method.Based on the shadow detection results,we decompose the input video into overlapped 2D patches and find the correspondences between the shadow and non-shadow regions via coherent patch matching.After that,we remove the shadow and recover information under normal illumination in the shadow regions using illumination transfer optimization.We finally process the shadow boundaries to get spatial-temporal coherent shadow-free video results.Specifically,the main research contents and contributions of this paper are follows:(1)We present a matting-based shadow detection method for videos,which can detect both static and moving cast shadows and works well for complex shadow video scenarios.(2)We propose a motion field-guided coherent patch matching approach which can efficiently compute the dense correspondences between shadow and non-shadow patches in the video based on the patch similarity metric.(3)We propose a novel video shadow removal algorithm based on illumination transfer optimization.And we introduce a video shadow boundary processing approach,which makes the shadow removed video transits naturally at the shadow boundaries.Besides,we keep good spatial-temporal coherence while removing the shadow.The main research contents of this paper are around the video shadow detection and removal,which forms elementary system of video shadow detection and removal.Besides,we validate the effectiveness of our algorithm through experiments on a variety of videos and apply our algorithm to video color transfer.Our algorithm can be applied into field such as video editing,augment and virtual reality etc.