Abstract:We present two deep unfolding neural networks for the simultaneous tasks of background subtraction and foreground detection in video. Unlike conventional neural networks based on deep feature extraction, we incorporate domain-knowledge models by considering a masked variation of the robust principal component analysis problem (RPCA). With this approach, we separate video clips into low-rank and sparse components, respectively corresponding to the backgrounds and foreground masks indicating the presence of moving objects. Our models, coined ROMAN-S and ROMAN-R, map the iterations of two alternating direction of multipliers methods (ADMM) to trainable convolutional layers, and the proximal operators are mapped to non-linear activation functions with trainable thresholds. This approach leads to lightweight networks with enhanced interpretability that can be trained on limited data. In ROMAN-S, the correlation in time of successive binary masks is controlled with side-information based on l1 - l1 minimization. ROMAN-R enhances the foreground detection by learning a dictionary of atoms to represent the moving foreground in a high-dimensional feature space and by using reweighted- l1 - l1 minimization. Experiments are conducted on both synthetic and real video datasets, for which we also include an analysis of the generalization to unseen clips. Comparisons are made with existing deep unfolding RPCA neural networks, which do not use a mask formulation for the foreground, and with a 3D U-Net baseline. Results show that our proposed models outperform other deep unfolding networks, as well as the untrained optimization algorithms. ROMAN-R, in particular, is competitive with the U-Net baseline for foreground detection, with the additional advantage of providing video backgrounds and requiring substantially fewer training parameters and smaller training sets.

A Tensor-Based Online RPCA Model for Compressive Background Subtraction

Foreground-Background Parallel Compression with Residual Encoding for Surveillance Video

Background Subtraction Based on Nonparametric Bayesian Estimation.

Background Subtraction via Fast Robust Matrix Completion

Background Subtraction Based on Modified Pulse Coupled Neural Network in Compressive Domain

An Improved Total Variation Regularized RPCA for Moving Object Detection with Dynamic Background

Dynamic Mode Decomposition For Robust Pca With Applications To Foreground/Background Subtraction In Video Streams And Multi-Resolution Analysis

Modeling Background from Compressed Video

Approximated Robust Principal Component Analysis for Improved General Scene Background Subtraction

An Efficient Background Reconstruction Based Coding Method for Surveillance Videos Captured by Moving Camera

Complex Background Subtraction by Pursuing Dynamic Spatio-Temporal Models

rSVDdpd: A Robust Scalable Video Surveillance Background Modelling Algorithm

Denoising-Based Turbo Message Passing for Compressed Video Background Subtraction

DeepPBM: Deep Probabilistic Background Model Estimation from Video Sequences

Masked-RPCA: Sparse and Low-rank Decomposition Under Overlaying Model and Application to Moving Object Detection

Background Subtraction with Dynamic Noise Sampling and Complementary Learning

Online Compressed Robust PCA.

Background subtraction for video sequence using deep neural network

A Multilayer-Based Framework for Online Background Subtraction with Freely Moving Cameras

Interpretable Neural Networks for Video Separation: Deep Unfolding RPCA With Foreground Masking

Detection of Moving Object in Dynamic Background Using Gaussian Max-Pooling and Segmentation Constrained RPCA