Abstract:Video Moment Retrieval (VMR) aims to retrieve temporal segments in untrimmed videos corresponding to a given language query by constructing cross-modal alignment strategies. However, these existing strategies are often sub-optimal since they ignore the modality imbalance problem, \textit{i.e.}, the semantic richness inherent in videos far exceeds that of a given limited-length sentence. Therefore, in pursuit of better alignment, a natural idea is enhancing the video modality to filter out query-irrelevant semantics, and enhancing the text modality to capture more segment-relevant knowledge. In this paper, we introduce Modal-Enhanced Semantic Modeling (MESM), a novel framework for more balanced alignment through enhancing features at two levels. First, we enhance the video modality at the frame-word level through word reconstruction. This strategy emphasizes the portions associated with query words in frame-level features while suppressing irrelevant parts. Therefore, the enhanced video contains less redundant semantics and is more balanced with the textual modality. Second, we enhance the textual modality at the segment-sentence level by learning complementary knowledge from context sentences and ground-truth segments. With the knowledge added to the query, the textual modality thus maintains more meaningful semantics and is more balanced with the video modality. By implementing two levels of MESM, the semantic information from both modalities is more balanced to align, thereby bridging the modality gap. Experiments on three widely used benchmarks, including the out-of-distribution settings, show that the proposed framework achieves a new start-of-the-art performance with notable generalization ability (e.g., 4.42% and 7.69% average gains of R1@0.7 on Charades-STA and Charades-CG). The code will be available at <a class="link-external link-https" href="https://github.com/lntzm/MESM" rel="external noopener nofollow">this https URL</a>.

MAC-REALM: A Video Content Feature Extraction and Modelling Framework

End-to-end Semantic-centric Video-based Multimodal Affective Computing

Real-Time And Distributed Av Content Analysis System For Consumer Electronics Networks

Multimodal Salient Objects: General Building Blocks Of Semantic Video Concepts

MACAROON: Training Vision-Language Models To Be Your Engaged Partners

The Anatomy of a Modular System for Media Content Analysis

Modular Action Concept Grounding in Semantic Video Prediction

Macross: Urban Dynamics Modeling based on Metapath Guided Cross-Modal Embedding

A Statistics-Based Method For Video Semantic Analysis

Macaw-LLM: Multi-Modal Language Modeling with Image, Audio, Video, and Text Integration

Mars and its applications to MPEG-7

GenMAC: Compositional Text-to-Video Generation with Multi-Agent Collaboration

MacFormer: Semantic Segmentation with Fine Object Boundaries

MAC: A unified framework boosting low resource automatic speech recognition

A Hybrid Approach to Video Retrieval in a Generic Video Management and Application Processing Framework

MATF: main-auxiliary transformer fusion for multi-modal sentiment analysis

MACQ: A Holistic View of Model Acquisition Techniques

VideoMAC: Video Masked Autoencoders Meet ConvNets

Towards Balanced Alignment: Modal-Enhanced Semantic Modeling for Video Moment Retrieval

MACE: Mass Concept Erasure in Diffusion Models