Abstract:Recently, memory-based networks have achieved promising performance for video object segmentation (VOS). However, existing methods still suffer from unsatisfactory segmentation accuracy and inferior efficiency. The reasons are mainly twofold: 1) during memory construction, the inflexible memory storage mechanism results in a weak discriminative ability for similar appearances in complex scenarios, leading to video-level temporal redundancy, and 2) during memory reading, matching robustness and memory retrieval accuracy decrease as the number of video frames increases. To address these challenges, we propose an adaptive sparse memory network (ASM) that efficiently and effectively performs VOS by sparsely leveraging previous guidance while attending to key information. Specifically, we design an adaptive sparse memory constructor (ASMC) to adaptively memorize informative past frames according to dynamic temporal changes in video frames. Furthermore, we introduce an attentive local memory reader (ALMR) to quickly retrieve relevant information using a subset of memory, thereby reducing frame-level redundant computation and noise in a simpler and more convenient manner. To prevent key features from being discarded by the subset of memory, we further propose a novel attentive local feature aggregation (ALFA) module, which preserves useful cues by selectively aggregating discriminative spatial dependence from adjacent frames, thereby effectively increasing the receptive field of each memory frame. Extensive experiments demonstrate that our model achieves state-of-the-art performance with real-time speed on six popular VOS benchmarks. Furthermore, our ASM can be applied to existing memory-based methods as generic plugins to achieve significant performance improvements. More importantly, our method exhibits robustness in handling sparse videos with low frame rates.

Addressing Issues with Working Memory in Video Object Segmentation

Learning Quality-aware Dynamic Memory for Video Object Segmentation

Fast Real-Time Video Object Segmentation with a Tangled Memory Network

Adaptive Memory Management for Video Object Segmentation

Learning Position and Target Consistency for Memory-based Video Object Segmentation

RMem: Restricted Memory Banks Improve Video Object Segmentation

Adaptive Sparse Memory Networks for Efficient and Robust Video Object Segmentation

Robust and Efficient Memory Network for Video Object Segmentation

Modulated Memory Network for Video Object Segmentation

Joint Modeling of Feature, Correspondence, and a Compressed Memory for Video Object Segmentation

Beyond Appearance: Multi-Frame Spatio-Temporal Context Memory Networks for Efficient and Robust Video Object Segmentation

Temporally Consistent Referring Video Object Segmentation with Hybrid Memory

Efficient and Robust Video Object Segmentation Through Isogenous Memory Sampling and Frame Relation Mining

Memory-Efficient Continual Learning Object Segmentation for Long Video

Dual Temporal Memory Network for Efficient Video Object Segmentation

Video Object Segmentation with Dynamic Query Modulation

Memory Matching is not Enough: Jointly Improving Memory Matching and Decoding for Video Object Segmentation

READMem: Robust Embedding Association for a Diverse Memory in Unconstrained Video Object Segmentation

Adaptive Selection of Reference Frames for Video Object Segmentation.

HFVOS: History-Future Integrated Dynamic Memory for Video Object Segmentation

Efficient Regional Memory Network for Video Object Segmentation