Abstract:Recently, memory-based networks have achieved promising performance for video object segmentation (VOS). However, existing methods still suffer from unsatisfactory segmentation accuracy and inferior efficiency. The reasons are mainly twofold: 1) during memory construction, the inflexible memory storage mechanism results in a weak discriminative ability for similar appearances in complex scenarios, leading to video-level temporal redundancy, and 2) during memory reading, matching robustness and memory retrieval accuracy decrease as the number of video frames increases. To address these challenges, we propose an adaptive sparse memory network (ASM) that efficiently and effectively performs VOS by sparsely leveraging previous guidance while attending to key information. Specifically, we design an adaptive sparse memory constructor (ASMC) to adaptively memorize informative past frames according to dynamic temporal changes in video frames. Furthermore, we introduce an attentive local memory reader (ALMR) to quickly retrieve relevant information using a subset of memory, thereby reducing frame-level redundant computation and noise in a simpler and more convenient manner. To prevent key features from being discarded by the subset of memory, we further propose a novel attentive local feature aggregation (ALFA) module, which preserves useful cues by selectively aggregating discriminative spatial dependence from adjacent frames, thereby effectively increasing the receptive field of each memory frame. Extensive experiments demonstrate that our model achieves state-of-the-art performance with real-time speed on six popular VOS benchmarks. Furthermore, our ASM can be applied to existing memory-based methods as generic plugins to achieve significant performance improvements. More importantly, our method exhibits robustness in handling sparse videos with low frame rates.

Memory Aggregated CFBI+ for Interactive Video Object Segmentation

Learning Quality-aware Dynamic Memory for Video Object Segmentation

Fast Real-Time Video Object Segmentation with a Tangled Memory Network

Memory Aggregation Networks for Efficient Interactive Video Object Segmentation

Collaborative Video Object Segmentation by Foreground-Background Integration

Beyond Appearance: Multi-Frame Spatio-Temporal Context Memory Networks for Efficient and Robust Video Object Segmentation

Video object segmentation via couple streams and feature memory

Efficient and Robust Video Object Segmentation Through Isogenous Memory Sampling and Frame Relation Mining

Global Spectral Filter Memory Network for Video Object Segmentation.

Efficient video object segmentation based on Gaussian mixture model and Markov random field

Video Object Segmentation via Structural Feature Reconfiguration

Motion-Appearance Interactive Encoding for Object Segmentation in Unconstrained Videos

Multiresolution Segmentation of Video Objects in the Compression Domain

Robust and Efficient Memory Network for Video Object Segmentation

Learning effective feature representation for video object segmentation via memory

Bilateral Temporal Re-Aggregation for Weakly-supervised Video Object Segmentation

Video Object Segmentation with Dynamic Memory Networks and Adaptive Object Alignment.

Memory Matching is not Enough: Jointly Improving Memory Matching and Decoding for Video Object Segmentation

Adaptive Selection of Reference Frames for Video Object Segmentation.

Adaptive Sparse Memory Networks for Efficient and Robust Video Object Segmentation

Alignment Before Aggregation: Trajectory Memory Retrieval Network for Video Object Segmentation