Abstract:We present two deep unfolding neural networks for the simultaneous tasks of background subtraction and foreground detection in video. Unlike conventional neural networks based on deep feature extraction, we incorporate domain-knowledge models by considering a masked variation of the robust principal component analysis problem (RPCA). With this approach, we separate video clips into low-rank and sparse components, respectively corresponding to the backgrounds and foreground masks indicating the presence of moving objects. Our models, coined ROMAN-S and ROMAN-R, map the iterations of two alternating direction of multipliers methods (ADMM) to trainable convolutional layers, and the proximal operators are mapped to non-linear activation functions with trainable thresholds. This approach leads to lightweight networks with enhanced interpretability that can be trained on limited data. In ROMAN-S, the correlation in time of successive binary masks is controlled with side-information based on l1 - l1 minimization. ROMAN-R enhances the foreground detection by learning a dictionary of atoms to represent the moving foreground in a high-dimensional feature space and by using reweighted- l1 - l1 minimization. Experiments are conducted on both synthetic and real video datasets, for which we also include an analysis of the generalization to unseen clips. Comparisons are made with existing deep unfolding RPCA neural networks, which do not use a mask formulation for the foreground, and with a 3D U-Net baseline. Results show that our proposed models outperform other deep unfolding networks, as well as the untrained optimization algorithms. ROMAN-R, in particular, is competitive with the U-Net baseline for foreground detection, with the additional advantage of providing video backgrounds and requiring substantially fewer training parameters and smaller training sets.

Masked-RPCA: Sparse and Low-rank Decomposition Under Overlaying Model and Application to Moving Object Detection

Moving Object Detection in Dynamic Environment Via Weighted Low-Rank Structured Sparse RPCA and Kalman Filtering

Moving Object Detection Based on On-Line Block-Robust Principal Component Analysis Decomposition

An Improved Total Variation Regularized RPCA for Moving Object Detection with Dynamic Background

Moving Object Detection by Patch Dividing Through Low-Rank Framework

Dynamic Mode Decomposition For Robust Pca With Applications To Foreground/Background Subtraction In Video Streams And Multi-Resolution Analysis

Detection of Moving Object in Dynamic Background Using Gaussian Max-Pooling and Segmentation Constrained RPCA

Joint Video Frame Set Division and Low-Rank Decomposition for Background Subtraction

Moving Object Detection Via Low-Rank Total Variation Regularization

COROLA: A Sequential Solution to Moving Object Detection Using Low-rank Approximation

On the Applications of Robust PCA in Image and Video Processing

Learned Masked Robust Principal Component Analysis Model for Infrared Small Target Detection

Interpretable Neural Networks for Video Separation: Deep Unfolding RPCA With Foreground Masking

Multi-ROI Background Modeling Method Suitable for HD Monitor Video

Background subtraction based on low-rank and structured sparse decomposition.

Moving Target Detection by Robust PCA in the Topological Space of Low-Rank Matrices

Background Separation Based on Dual-Weighted Robust Principle Component Analysis

Salient Motion Detection Using Proximal Robust Principal Component Analysis Method

rSVDdpd: A Robust Scalable Video Surveillance Background Modelling Algorithm

Robust hybrid technique for moving object detection and tracking using cartoon features and fast PCP

A Novel Tensor Robust PCA Approach for Background Subtraction from Compressive Measurements.