Abstract:Recently, memory-based networks have achieved promising performance for video object segmentation (VOS). However, existing methods still suffer from unsatisfactory segmentation accuracy and inferior efficiency. The reasons are mainly twofold: 1) during memory construction, the inflexible memory storage mechanism results in a weak discriminative ability for similar appearances in complex scenarios, leading to video-level temporal redundancy, and 2) during memory reading, matching robustness and memory retrieval accuracy decrease as the number of video frames increases. To address these challenges, we propose an adaptive sparse memory network (ASM) that efficiently and effectively performs VOS by sparsely leveraging previous guidance while attending to key information. Specifically, we design an adaptive sparse memory constructor (ASMC) to adaptively memorize informative past frames according to dynamic temporal changes in video frames. Furthermore, we introduce an attentive local memory reader (ALMR) to quickly retrieve relevant information using a subset of memory, thereby reducing frame-level redundant computation and noise in a simpler and more convenient manner. To prevent key features from being discarded by the subset of memory, we further propose a novel attentive local feature aggregation (ALFA) module, which preserves useful cues by selectively aggregating discriminative spatial dependence from adjacent frames, thereby effectively increasing the receptive field of each memory frame. Extensive experiments demonstrate that our model achieves state-of-the-art performance with real-time speed on six popular VOS benchmarks. Furthermore, our ASM can be applied to existing memory-based methods as generic plugins to achieve significant performance improvements. More importantly, our method exhibits robustness in handling sparse videos with low frame rates.

Multi-Involution Memory Network for Unsupervised Video Object Segmentation

Fast Real-Time Video Object Segmentation with a Tangled Memory Network

Learning Spatiotemporal Relationships with a Unified Framework for Video Object Segmentation

Learning Quality-aware Dynamic Memory for Video Object Segmentation

Implicit Motion-Compensated Network for Unsupervised Video Object Segmentation

Unsupervised Online Video Object Segmentation with Motion Property Understanding

Learning Motion and Temporal Cues for Unsupervised Video Object Segmentation

Dual Temporal Memory Network for Efficient Video Object Segmentation

Robust and Efficient Memory Network for Video Object Segmentation

Video Object Segmentation using Space-Time Memory Networks

Reciprocal Transformations for Unsupervised Video Object Segmentation

MUNet: Motion uncertainty-aware semi-supervised video object segmentation

Adaptive Sparse Memory Networks for Efficient and Robust Video Object Segmentation

Modulated Memory Network for Video Object Segmentation

Tsanet: Temporal and Scale Alignment for Unsupervised Video Object Segmentation

Learning Memory Propagation And Matching For Semi-Supervised Video Object Segmentation

Enhanced Memory Network for Video Segmentation

Learning Video Object Segmentation with Visual Memory

Memory Aggregation Networks for Efficient Interactive Video Object Segmentation

Video object segmentation via couple streams and feature memory

Efficient and Robust Video Object Segmentation Through Isogenous Memory Sampling and Frame Relation Mining