Foreground-Background Separation From Video Clips Via Motion-Assisted Matrix Restoration

Separation of video streams into foreground and background components is a useful and important technique in video analysis, making recognition, classification and scene analysis more efficient.In this paper, we propose a motion-assisted matrix restoration(MAMR) model for foreground-background separation from video clips. In the proposed MAMR model, the backgrounds across frames are modeled by a low-rank matrix, while the foreground objects are modeled by a sparse matrix. To facilitate efficient foreground-background separation, a dense motion field is estimated for each frame, and mapped into a weighting matrix which indicates the likelihood that each pixel belongs to the background. Anchor frames are selected in the dense motion estimation to overcome the difficulty of detecting slowlymoving objects and camouflages.The MAMR model is solved by the alternating direction method under the augmented Lagrangian multiplier(ALM) framework. Then the foreground is computed by the background subtraction technique using the recovered background image. In addition, we extend our model to a robust MAMR model(RMAMR) which is robust to noise for practical applications.In experiments, we first investigate how the motion-to-weight mapping affects the performance; then we compare our MAMR method with other state-of-the-art methods on challenging datasets, and report experimental results for both background extraction and foreground detection; finally our RMAMR method is tested on datasets with synthetic dense noise. The evaluations on these datasets demonstrate our method is quite versatile for surveillance videos with lighting variations, camouflages, different types of motions, and outperforms many other state-of-the-art methods.