Abstract:Extracting semantically consistent representations from multi-modal data helps computers understand the human world more comprehensively. Visual-semantic matching, as one of the fundamental tasks for multi-modal learning, attracts continuous attention. Recent research makes unflagging endeavors to enhance the matching performance, but sometimes at the expense of overlooking the delicate balance between efficiency and effectiveness. In this paper, we aim to address this dilemma through a newly proposed attention-mechanism-based architecture. To ensure optimal effectiveness, we adopt a more advanced Transformer Encoder (TE) as our basic model and introduce two significant ameliorations to tailor it for the visual-semantic matching task. Initially, we incorporate fine-grained supervision into the classic TE, allowing our model to capture sophisticated correspondences between different modalities. Subsequently, we employ a dynamic attention-evolving strategy to selectively pass useful information and strengthen the attention pattern consistency between adjacent TE blocks. To maintain efficiency, we propose a novel Select & Re-rank strategy that enables the model to ignore redundant information. This approach significantly reduces the computational cost and increases the matching speed with relatively minimal performance degradation. The proposed architecture can gradually capture and reorganize useful information from inter-modality as well as intra-modality under the supervision of both fine-grained and global similarity, which leads to more comprehensive and discriminative embeddings. Experiments on two benchmark datasets show that the proposed method achieves competitive results in terms of both efficiency and effectiveness.

Embrace Smaller Attention: Efficient Cross-Modal Matching with Dual Gated Attention Fusion

Fine-Grained Cross-Modal Retrieval with Triple-Streamed Memory Fusion Transformer Encoder

Select & Re-Rank: Effectively and Efficiently Matching Multimodal Data with Dynamically Evolving Attention

Pre-gating and contextual attention gate - A new fusion method for multi-modal data tasks

Cross-Attention is Not Enough: Incongruity-Aware Dynamic Hierarchical Fusion for Multimodal Affect Recognition

Two-Stream Video Classification with Cross-Modality Attention

Multimodal Fusion Method Based on Self-Attention Mechanism

Cross‐modal retrieval with dual multi‐angle self‐attention

Iterative graph attention memory network for cross-modal retrieval

Bridging the gap: dual perception attention and local-global similarity fusion for cross-modal image-text matching

Multimodal Sentiment Analysis Based on Cross-Modal Attention and Gated Cyclic Hierarchical Fusion Networks

ICAFusion: Iterative cross-attention guided feature fusion for multispectral object detection

Efficient Multi-Scale Attention Module with Cross-Spatial Learning

Object fusion tracking for RGB-T images via channel swapping and modal mutual attention

MCFusion: infrared and visible image fusion based multiscale receptive field and cross-modal enhanced attention mechanism

Cross-Modality Attentive Feature Fusion for Object Detection in Multispectral Remote Sensing Imagery

A Cross-scale Iterative Attentional Adversarial Fusion Network for Infrared and Visible Images

Cross-Attention is Not Always Needed: Dynamic Cross-Attention for Audio-Visual Dimensional Emotion Recognition

Attentional Feature Fusion

CrossFuse: A Novel Cross Attention Mechanism based Infrared and Visible Image Fusion Approach

Enhancing target detection accuracy through cross-modal spatial perception and dual-modality fusion