Efficient Video Object Segmentation via Modulated Cross-Attention Memory

Abdelrahman Shaker,Syed Talal Wasim,Martin Danelljan,Salman Khan,Ming-Hsuan Yang,Fahad Shahbaz Khan

2024-09-26

Abstract:Recently, transformer-based approaches have shown promising results for semi-supervised video object segmentation. However, these approaches typically struggle on long videos due to increased GPU memory demands, as they frequently expand the memory bank every few frames. We propose a transformer-based approach, named MAVOS, that introduces an optimized and dynamic long-term modulated cross-attention (MCA) memory to model temporal smoothness without requiring frequent memory expansion. The proposed MCA effectively encodes both local and global features at various levels of granularity while efficiently maintaining consistent speed regardless of the video length. Extensive experiments on multiple benchmarks, LVOS, Long-Time Video, and DAVIS 2017, demonstrate the effectiveness of our proposed contributions leading to real-time inference and markedly reduced memory demands without any degradation in segmentation accuracy on long videos. Compared to the best existing transformer-based approach, our MAVOS increases the speed by 7.6x, while significantly reducing the GPU memory by 87% with comparable segmentation performance on short and long video datasets. Notably on the LVOS dataset, our MAVOS achieves a J&F score of 63.3% while operating at 37 frames per second (FPS) on a single V100 GPU. Our code and models will be publicly available at: <a class="link-external link-https" href="https://github.com/Amshaker/MAVOS" rel="external noopener nofollow">this https URL</a>.

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

The problem that this paper attempts to solve is the problems of excessive GPU memory requirements and a significant speed drop encountered by existing Transformer - based Video Object Segmentation (VOS) methods when processing long - videos. Specifically:

1. **Memory Expansion Problem**: Existing Transformer - based VOS methods usually need to expand the memory bank every few frames when processing long - videos, resulting in an increasing demand for GPU memory and ultimately unable to meet the requirements of long - video processing.
2. **Speed Drop Problem**: As the length of the video increases, the speed of these methods will drop significantly, for example, from 30 FPS to 5 FPS, which makes it difficult for them to achieve real - time processing.

To solve these problems, the author proposes a new Transformer - based method named MAVOS (Modulated Cross - Attention Memory for Video Object Segmentation). This method introduces an optimized dynamic long - term Modulated Cross - Attention (MCA) memory mechanism, which can effectively encode local and global features without frequently expanding the memory, and maintain a consistent speed and GPU memory consumption, thereby achieving efficient and real - time video object segmentation.

### Main Contributions
- **Optimized Long - term Modulated Cross - Attention (MCA) Memory**: MAVOS avoids the need for frequent memory expansion by introducing the MCA memory mechanism and can maintain consistent GPU memory consumption and speed when processing long - videos.
- **Efficient Temporal Smoothness Modeling**: The MCA memory mechanism can effectively capture temporal and spatial information in past frames and ensure the accurate propagation of target information.
- **Superior Performance**: Experimental results show that MAVOS performs excellently on multiple benchmark datasets (such as LVOS, Long - Time Video and DAVIS 2017). It not only achieves real - time inference but also significantly reduces GPU memory requirements while maintaining segmentation accuracy comparable to existing best methods.

Through these improvements, MAVOS can provide more efficient and stable performance when processing long - videos and is suitable for application scenarios that require real - time processing and low memory footprint.