Abstract:Recently, memory-based networks have achieved promising performance for video object segmentation (VOS). However, existing methods still suffer from unsatisfactory segmentation accuracy and inferior efficiency. The reasons are mainly twofold: 1) during memory construction, the inflexible memory storage mechanism results in a weak discriminative ability for similar appearances in complex scenarios, leading to video-level temporal redundancy, and 2) during memory reading, matching robustness and memory retrieval accuracy decrease as the number of video frames increases. To address these challenges, we propose an adaptive sparse memory network (ASM) that efficiently and effectively performs VOS by sparsely leveraging previous guidance while attending to key information. Specifically, we design an adaptive sparse memory constructor (ASMC) to adaptively memorize informative past frames according to dynamic temporal changes in video frames. Furthermore, we introduce an attentive local memory reader (ALMR) to quickly retrieve relevant information using a subset of memory, thereby reducing frame-level redundant computation and noise in a simpler and more convenient manner. To prevent key features from being discarded by the subset of memory, we further propose a novel attentive local feature aggregation (ALFA) module, which preserves useful cues by selectively aggregating discriminative spatial dependence from adjacent frames, thereby effectively increasing the receptive field of each memory frame. Extensive experiments demonstrate that our model achieves state-of-the-art performance with real-time speed on six popular VOS benchmarks. Furthermore, our ASM can be applied to existing memory-based methods as generic plugins to achieve significant performance improvements. More importantly, our method exhibits robustness in handling sparse videos with low frame rates.

Video Object Segmentation with Episodic Graph Memory Networks.

Video Object Segmentation with Episodic Graph Memory Networks

Fast Real-Time Video Object Segmentation with a Tangled Memory Network

Learning Quality-aware Dynamic Memory for Video Object Segmentation

Dual Temporal Memory Network for Efficient Video Object Segmentation

Dual temporal memory network with high-order spatio-temporal graph learning for video object segmentation

Video Object Segmentation using Space-Time Memory Networks

Learning Spatiotemporal Relationships with a Unified Framework for Video Object Segmentation

TransVOS: Video Object Segmentation with Transformers

Learning Video Object Segmentation with Visual Memory

Video object segmentation via couple streams and feature memory

Robust and Efficient Memory Network for Video Object Segmentation

LiDAR Video Object Segmentation with Dynamic Kernel Refinement

Space-time Reinforcement Network for Video Object Segmentation

Adaptive Sparse Memory Networks for Efficient and Robust Video Object Segmentation

CapsuleVOS: Semi-Supervised Video Object Segmentation Using Capsule Routing

Beyond Appearance: Multi-Frame Spatio-Temporal Context Memory Networks for Efficient and Robust Video Object Segmentation

Adaptive Memory Management for Video Object Segmentation

Hierarchical Memory Matching Network for Video Object Segmentation

Global Spectral Filter Memory Network for Video Object Segmentation.

Two-Stream Networks for Object Segmentation in Videos