Abstract:Recently, memory-based networks have achieved promising performance for video object segmentation (VOS). However, existing methods still suffer from unsatisfactory segmentation accuracy and inferior efficiency. The reasons are mainly twofold: 1) during memory construction, the inflexible memory storage mechanism results in a weak discriminative ability for similar appearances in complex scenarios, leading to video-level temporal redundancy, and 2) during memory reading, matching robustness and memory retrieval accuracy decrease as the number of video frames increases. To address these challenges, we propose an adaptive sparse memory network (ASM) that efficiently and effectively performs VOS by sparsely leveraging previous guidance while attending to key information. Specifically, we design an adaptive sparse memory constructor (ASMC) to adaptively memorize informative past frames according to dynamic temporal changes in video frames. Furthermore, we introduce an attentive local memory reader (ALMR) to quickly retrieve relevant information using a subset of memory, thereby reducing frame-level redundant computation and noise in a simpler and more convenient manner. To prevent key features from being discarded by the subset of memory, we further propose a novel attentive local feature aggregation (ALFA) module, which preserves useful cues by selectively aggregating discriminative spatial dependence from adjacent frames, thereby effectively increasing the receptive field of each memory frame. Extensive experiments demonstrate that our model achieves state-of-the-art performance with real-time speed on six popular VOS benchmarks. Furthermore, our ASM can be applied to existing memory-based methods as generic plugins to achieve significant performance improvements. More importantly, our method exhibits robustness in handling sparse videos with low frame rates.

Occlusion Tolerant Object Recognition Using Visual Memory Selection Model

Local Spatial Continuity Steered Sparse Representation for Occluded Face Recognition

Face recognition with contiguous occlusion using linear regression and level set method

Object-based Encoding in Visual Working Memory: Evidence from Memory-driven Attentional Capture

Object-based Encoding in Visual Working Memory: A Critical Revisit.

Object Detection Based on Saturation of Visual Perception

Brain Functional Representation of Highly Occluded Object Recognition

VMM: Viewpoint-based Memory Mechanism for Object Detection of Moving Sensors

Vehicle Recognition Model for Complex Scenarios Based on Human Memory Mechanism.

Addressing Issues with Working Memory in Video Object Segmentation

Adversarially Learning Occlusions by Backpropagation for Face Recognition

Improved Visual Recognition Memory Model Based on Grid Cells for Face Recognition

Modulated Memory Network for Video Object Segmentation

Sequential Amodal Segmentation via Cumulative Occlusion Learning

Occluded Video Instance Segmentation: A Benchmark

RMem: Restricted Memory Banks Improve Video Object Segmentation

Deep Visual Odometry with Adaptive Memory

Robust and Efficient Memory Network for Video Object Segmentation

Tracking-Assisted Object Detection with Event Cameras

Adaptive Sparse Memory Networks for Efficient and Robust Video Object Segmentation

Long short‐term memory‐based neural decoding of object categories evoked by natural images