Abstract:Extracting semantically consistent representations from multi-modal data helps computers understand the human world more comprehensively. Visual-semantic matching, as one of the fundamental tasks for multi-modal learning, attracts continuous attention. Recent research makes unflagging endeavors to enhance the matching performance, but sometimes at the expense of overlooking the delicate balance between efficiency and effectiveness. In this paper, we aim to address this dilemma through a newly proposed attention-mechanism-based architecture. To ensure optimal effectiveness, we adopt a more advanced Transformer Encoder (TE) as our basic model and introduce two significant ameliorations to tailor it for the visual-semantic matching task. Initially, we incorporate fine-grained supervision into the classic TE, allowing our model to capture sophisticated correspondences between different modalities. Subsequently, we employ a dynamic attention-evolving strategy to selectively pass useful information and strengthen the attention pattern consistency between adjacent TE blocks. To maintain efficiency, we propose a novel Select & Re-rank strategy that enables the model to ignore redundant information. This approach significantly reduces the computational cost and increases the matching speed with relatively minimal performance degradation. The proposed architecture can gradually capture and reorganize useful information from inter-modality as well as intra-modality under the supervision of both fine-grained and global similarity, which leads to more comprehensive and discriminative embeddings. Experiments on two benchmark datasets show that the proposed method achieves competitive results in terms of both efficiency and effectiveness.

Semantic Matching Model based on Layer-Wise Attention Pooling Network and Dynamic Feature Fusion Mechanism

Select & Re-Rank: Effectively and Efficiently Matching Multimodal Data with Dynamically Evolving Attention

Chinese Semantic Matching with Multi-granularity Alignment and Feature Fusion

Research of improving semantic image segmentation based on a feature fusion model

Legal Feature Enhanced Semantic Matching Network for Similar Case Matching

DABERT: Dual Attention Enhanced BERT for Semantic Matching.

Multi-scale Matching Networks for Semantic Correspondence

Improving Semantic Matching Through Dependency-Enhanced Pre-trained Model with Adaptive Fusion.

Research on semantic matching algorithm of BERT intelligent question answering system

Short Text Semantic Matching Model Based on Bert and Adversarial Network.

Leveraging Semantic and Lexical Matching to Improve the Recall of Document Retrieval Systems: A Hybrid Approach

Semantic Similarity Computing Model Based on Multi Model Fine-Grained Nonlinear Fusion

RI-Match: Integrating Both Representations and Interactions for Deep Semantic Matching.

Multi-Modal Fusion-Based Multi-Task Semantic Communication System

Deep Fusion LSTMs for Text Semantic Matching.

Frame-based Multi-level Semantics Representation for text matching

Lightweight Text Matching Method with Rich Features.

A Deep Fusion Matching Network Semantic Reasoning Model

A Deep Architecture for Semantic Matching with Multiple Positional Sentence Representations

Chinese long text similarity calculation of semantic progressive fusion based on Bert

Modeling Selective Feature Attention for Representation-based Siamese Text Matching