Abstract:Extracting semantically consistent representations from multi-modal data helps computers understand the human world more comprehensively. Visual-semantic matching, as one of the fundamental tasks for multi-modal learning, attracts continuous attention. Recent research makes unflagging endeavors to enhance the matching performance, but sometimes at the expense of overlooking the delicate balance between efficiency and effectiveness. In this paper, we aim to address this dilemma through a newly proposed attention-mechanism-based architecture. To ensure optimal effectiveness, we adopt a more advanced Transformer Encoder (TE) as our basic model and introduce two significant ameliorations to tailor it for the visual-semantic matching task. Initially, we incorporate fine-grained supervision into the classic TE, allowing our model to capture sophisticated correspondences between different modalities. Subsequently, we employ a dynamic attention-evolving strategy to selectively pass useful information and strengthen the attention pattern consistency between adjacent TE blocks. To maintain efficiency, we propose a novel Select & Re-rank strategy that enables the model to ignore redundant information. This approach significantly reduces the computational cost and increases the matching speed with relatively minimal performance degradation. The proposed architecture can gradually capture and reorganize useful information from inter-modality as well as intra-modality under the supervision of both fine-grained and global similarity, which leads to more comprehensive and discriminative embeddings. Experiments on two benchmark datasets show that the proposed method achieves competitive results in terms of both efficiency and effectiveness.

Dual Path Modeling for Semantic Matching by Perceiving Subtle Conflicts

Select & Re-Rank: Effectively and Efficiently Matching Multimodal Data with Dynamically Evolving Attention

Do Multi-Sense Embeddings Improve Natural Language Understanding?

DABERT: Dual Attention Enhanced BERT for Semantic Matching.

Dual Path Structural Contrastive Embeddings for Learning Novel Objects

Dual Semantic Relationship Attention Network for Image-Text Matching

Comateformer: Combined Attention Transformer for Semantic Sentence Matching

Improving Semantic Matching Through Dependency-Enhanced Pre-trained Model with Adaptive Fusion.

Multi-scale Matching Networks for Semantic Correspondence

Learning Dual Semantic Relations with Graph Attention for Image-Text Matching

Dual-path Rare Content Enhancement Network for Image and Text Matching

Dual-Path Feature Fusion Network for Semantic Segmentation of Remote Sensing Images

Dual-Path Feature Aware Network for Remote Sensing Image Semantic Segmentation

Model-Level Dual Learning.

SeaDATE: Remedy Dual-Attention Transformer with Semantic Alignment via Contrast Learning for Multimodal Object Detection

Short Text Semantic Matching Model Based on Bert and Adversarial Network.

Enhanced Text Matching Based on Semantic Transformation

Cross-modal Semantic Interference Suppression for image-text matching

Understanding Dark Scenes by Contrasting Multi-Modal Observations

Dual-oriented Disentangled Network with Counterfactual Intervention for Multimodal Intent Detection