Abstract:Recently, memory-based networks have achieved promising performance for video object segmentation (VOS). However, existing methods still suffer from unsatisfactory segmentation accuracy and inferior efficiency. The reasons are mainly twofold: 1) during memory construction, the inflexible memory storage mechanism results in a weak discriminative ability for similar appearances in complex scenarios, leading to video-level temporal redundancy, and 2) during memory reading, matching robustness and memory retrieval accuracy decrease as the number of video frames increases. To address these challenges, we propose an adaptive sparse memory network (ASM) that efficiently and effectively performs VOS by sparsely leveraging previous guidance while attending to key information. Specifically, we design an adaptive sparse memory constructor (ASMC) to adaptively memorize informative past frames according to dynamic temporal changes in video frames. Furthermore, we introduce an attentive local memory reader (ALMR) to quickly retrieve relevant information using a subset of memory, thereby reducing frame-level redundant computation and noise in a simpler and more convenient manner. To prevent key features from being discarded by the subset of memory, we further propose a novel attentive local feature aggregation (ALFA) module, which preserves useful cues by selectively aggregating discriminative spatial dependence from adjacent frames, thereby effectively increasing the receptive field of each memory frame. Extensive experiments demonstrate that our model achieves state-of-the-art performance with real-time speed on six popular VOS benchmarks. Furthermore, our ASM can be applied to existing memory-based methods as generic plugins to achieve significant performance improvements. More importantly, our method exhibits robustness in handling sparse videos with low frame rates.

RGB-D Video Object Segmentation Via Enhanced Multi-store Feature Memory

Fast Real-Time Video Object Segmentation with a Tangled Memory Network

Learning Quality-aware Dynamic Memory for Video Object Segmentation

LiDAR Video Object Segmentation with Dynamic Kernel Refinement

Weakly-Supervised RGBD Video Object Segmentation

Dual Temporal Memory Network for Efficient Video Object Segmentation

Robust and Efficient Memory Network for Video Object Segmentation

Video Object Segmentation with Dynamic Query Modulation

Video object segmentation via couple streams and feature memory

Learning Spatial-Semantic Features for Robust Video Object Segmentation

RGB-D Video Segmentation via Geodesic Spatio-Temporal Propagation

Beyond Appearance: Multi-Frame Spatio-Temporal Context Memory Networks for Efficient and Robust Video Object Segmentation

Recurrent Dynamic Embedding for Video Object Segmentation

Adaptive Sparse Memory Networks for Efficient and Robust Video Object Segmentation

Spatial-Temporal Multi-level Association for Video Object Segmentation

Object Detection and Tracking under Occlusion for Object-Level RGB-D Video Segmentation

Spatial-information Guided Adaptive Context-aware Network for Efficient RGB-D Semantic Segmentation

Video Object Segmentation using Space-Time Memory Networks

Dual temporal memory network with high-order spatio-temporal graph learning for video object segmentation

Region Aware Video Object Segmentation With Deep Motion Modeling

Video Object Segmentation via Structural Feature Reconfiguration