Abstract:Massive numbers of new images are uploaded to the internet every day. However, existing cross-modal retrieval (CMR) approaches struggle to accommodate this continuously growing data. The prevalent practice involves periodically retraining or fine-tuning a new model based on the accumulated data, which in turn invalidates billions of indexed features extracted by the previous model and incurs another substantial computational cost to extract new features for the entire data archive. Is it possible to develop a retrieval model that effectively captures the knowledge of upcoming sessions while preserving the discriminative power of features extracted in previous sessions? In this paper, we propose an online continual learning setup, OC-CMR, to formalize the data-incremental growth challenge faced by cross-modal retrieval systems. It consists of two key settings: 1) Similar to the real-world scenarios, the streaming multi-modal data arrives once per session; 2) Consider the computational costs, each instance of archived data has its feature extracted only once and by its corresponding model in its session. Based on our OC-CMR, we perform in-depth evaluations of state-of-the-art cross-modal retrieval methods and observe that they suffer from representational shift and collapse due to the catastrophic forgetting. To address this issue, we propose the Continual Cross-Modal Retrieval (C2MR) approach, which learns a shared common space not only across modalities but also sessions and maintains relationships between samples from distinct sessions via cross-modal relational coherence and semantic representation coordination. We construct two new benchmarks by adapting MS-COCO and Flickr30K datasets to the OC-CMR setting, providing a more challenging evaluation framework for CMR tasks. Experimental results demonstrate that our method effectively alleviates forgetting and significantly outperforms combinations of previous arts in cross-modal retrieval and continual learning.

Fine-Grained Cross-Modal Retrieval with Triple-Streamed Memory Fusion Transformer Encoder

Embrace Smaller Attention: Efficient Cross-Modal Matching with Dual Gated Attention Fusion

Coarse-to-fine dual-level attention for video-text cross modal retrieval

Multi-grained Encoding and Joint Embedding Space Fusion for Video and Text Cross-Modal Retrieval

Feature Fusion Based on Transformer for Cross-modal Retrieval

Multi-modal Fine-grained Retrieval with Local and Global Cross-Attention.

Attention is not Enough: Mitigating the Distribution Discrepancy in Asynchronous Multimodal Sequence Fusion

Deep Memory Network for Cross-Modal Retrieval

Memory Enhanced Embedding Learning for Cross-Modal Video-Text Retrieval

Iterative graph attention memory network for cross-modal retrieval

A Fusion Encoder with Multi-Task Guidance for Cross-Modal Text–Image Retrieval in Remote Sensing

Multi-Modal Memory Enhancement Attention Network for Image-Text Matching

Cross-Modal Message Passing for Two-stream Fusion

C2MR: Continual Cross-Modal Retrieval for Streaming Multi-modal Data

Hierarchical Attention-Based Multimodal Fusion for Video Captioning

Modality-specific Adaptive Scaling and Attention Network for Cross-Modal Retrieval

Two-Stream Video Classification with Cross-Modality Attention

Multimodal Fusion with Co-attention Mechanism

Multimodal Fusion Method Based on Self-Attention Mechanism

Research on Video Retrieval Technology based on Multimodal Fusion and Attention Mechanism

Multi-Encoder Learning and Stream Fusion for Transformer-Based End-to-End Automatic Speech Recognition