Abstract:Extracting semantically consistent representations from multi-modal data helps computers understand the human world more comprehensively. Visual-semantic matching, as one of the fundamental tasks for multi-modal learning, attracts continuous attention. Recent research makes unflagging endeavors to enhance the matching performance, but sometimes at the expense of overlooking the delicate balance between efficiency and effectiveness. In this paper, we aim to address this dilemma through a newly proposed attention-mechanism-based architecture. To ensure optimal effectiveness, we adopt a more advanced Transformer Encoder (TE) as our basic model and introduce two significant ameliorations to tailor it for the visual-semantic matching task. Initially, we incorporate fine-grained supervision into the classic TE, allowing our model to capture sophisticated correspondences between different modalities. Subsequently, we employ a dynamic attention-evolving strategy to selectively pass useful information and strengthen the attention pattern consistency between adjacent TE blocks. To maintain efficiency, we propose a novel Select & Re-rank strategy that enables the model to ignore redundant information. This approach significantly reduces the computational cost and increases the matching speed with relatively minimal performance degradation. The proposed architecture can gradually capture and reorganize useful information from inter-modality as well as intra-modality under the supervision of both fine-grained and global similarity, which leads to more comprehensive and discriminative embeddings. Experiments on two benchmark datasets show that the proposed method achieves competitive results in terms of both efficiency and effectiveness.

Going Beyond Multi-Task Dense Prediction with Synergy Embedding Models

Select & Re-Rank: Effectively and Efficiently Matching Multimodal Data with Dynamically Evolving Attention

MUCH: Mutual Coupling Enhancement of Scene Recognition and Dense Captioning

Learning from Semantically Dependent Multi-Tasks

UNITE: Multitask Learning with Sufficient Feature for Dense Prediction

Multi-Task Visual Semantic Embedding Network for Image-Text Retrieval

Semantics-Depth-Symbiosis: Deeply Coupled Semi-Supervised Learning of Semantics and Depth

Mutual Complementarity: Multi-Modal Enhancement Semantic Learning for Micro-Video Scene Recognition

Predicting Cross-Domain Collaboration Using Multi-Task Learning

Meta Multi-Task Learning for Sequence Modeling.

IMENet: Joint 3D Semantic Scene Completion and 2D Semantic Segmentation through Iterative Mutual Enhancement

MTMamba++: Enhancing Multi-Task Dense Scene Understanding via Mamba-Based Decoders

On the Consensus of Synchronous Temporal and Spatial Views: A Novel Multimodal Deep Learning Method for Social Video Prediction

MTMamba: Enhancing Multi-Task Dense Scene Understanding by Mamba-Based Decoders

Improving Multiple Dense Prediction Performances by Exploiting Inter-Task Synergies for Neuromorphic Vision Sensors

Cross Scene Prediction via Modeling Dynamic Correlation using Latent Space Shared Auto-Encoders

Bi-level Dynamic Learning for Jointly Multi-modality Image Fusion and Beyond

Multi-Task Collaborative Network for Joint Referring Expression Comprehension and Segmentation

Multi-Timescale Context Encoding for Scene Parsing Prediction

IIMT-net: Poly-1 Weights Balanced Multi-Task Network for Semantic Segmentation and Depth Estimation Using Interactive Information

Efficient Computation Sharing for Multi-Task Visual Scene Understanding