Abstract:Video moment retrieval (VMR) involves localizing video segments semantically aligned with given queries within videos. Despite the development of numerous methods for VMR in recent years, there remains a need to better incorporate fine-grained modality relation-aware information both in intra-modality and cross-modality. To address these challenges, we propose a Fine-grained Modality Relation-Aware Network (FMRN) tailored for the video moment retrieval task. FMRN effectively explores fine-grained modality relation-aware information within text queries, videos, and proposals. Our approach begins with a semantic graph encoder to capture deep semantic relations in intra-modality. Besides, we introduce a novel fine-grained cross-modality interaction module comprising a cross-similarity weighting module, an intra-modality weighting module, and an adaptive fusion module. These components comprehensively exploit fine-grained relation information within intra-modality and cross-modality contexts. Specifically, the cross-similarity weighting module leverages similarities between text queries and video snippets, as well as between videos and query words. The intra-modality weighting module determines the importance of words and snippets, while the adaptive fusion module combines cross-similarity weighting and intra-modality weighting. Additionally, we design a proposal relation module to enhance retrieval by capturing fine-grained proposals-relation information in videos. Extensive experiments demonstrate that the proposed method can outperform all state-of-the-art methods on the TACoS dataset and obtain comparable results on the Charades-STA and ActivityNet-Captions datasets. Compared with MCMN (TCSVT2024) and DPHANet (TMM2024), FMRN can achieve average improvements of 3.61 % and 5.44 % on the TACoS dataset, respectively 1 .

A multi-modal fusion approach for measuring web video relatedness

Concept-Driven Multi-Modality Fusion for Video Search

Dynamic Multimodal Fusion in Video Search

Online Video Recommendation Based on Multimodal Fusion and Relevance Feedback

Multi-grained Encoding and Joint Embedding Space Fusion for Video and Text Cross-Modal Retrieval

A Multi-modal Clustering Method for Web Videos

Multimodal feature fusion based on object relation for video captioning

Multi-modality Web Video Categorization

Video Visual Relation Detection Via Multi-modal Feature Fusion

Video Retrieval Model Based on Multimodal Information Fusion

A Novel Video Searching Model Based on Ontology Inference and Multimodal Information Fusion.

A Multi-interaction Model with Cross-Branch Feature Fusion for Video-Text Retrieval.

Deep Relationship Analysis in Video with Multimodal Feature Fusion

Cross-modal Search Method of Technology Video based on Adversarial Learning and Feature Fusion

Multimodal Fusion for Video Search Reranking

What Makes the Difference? an Empirical Comparison of Fusion Strategies for Multimodal Language Analysis.

Fusion of Multimodal Embeddings for Ad-Hoc Video Search

What Matters: Attentive and Relational Feature Aggregation Network for Video-Text Retrieval

Affective Video Content Analysis Based on Multimodal Data Fusion in Heterogeneous Networks.

Fine-Grained Modality Relation-Aware Network for Video Moment Retrieval