Abstract:Temporal grounding is the task of locating a specific segment from an untrimmed video according to a query sentence. This task has achieved significant momentum in the computer vision community as it enables activity grounding beyond pre-defined activity classes by utilizing the semantic diversity of natural language descriptions. The semantic diversity is rooted in the principle of compositionality in linguistics, where novel semantics can be systematically described by combining known words in novel ways (compositional generalization). However, existing temporal grounding datasets are not carefully designed to evaluate the compositional generalizability. To systematically benchmark the compositional generalizability of temporal grounding models, we introduce a new Compositional Temporal Grounding task and construct two new dataset splits, i.e., Charades-CG and ActivityNet-CG. We empirically find that they fail to generalize to queries with novel combinations of seen words. We argue that the inherent compositional structure (i.e., composition constituents and their relationships) inside the videos and language is the crucial factor to achieve compositional generalization. Based on this insight, we propose a variational cross-graph reasoning framework that explicitly decomposes video and language into hierarchical semantic graphs, respectively, and learns fine-grained semantic correspondence between the two graphs. Meanwhile, we introduce a novel adaptive structured semantics learning approach to derive the structure-informed and domain-generalizable graph representations, which facilitate the fine-grained semantic correspondence reasoning between the two graphs. To further evaluate the understanding of the compositional structure, we also introduce a more challenging setting, where one of the components in the novel composition is unseen. This requires more sophisticated understanding of the compositional structure to infer the potential semantics of the unseen word based on the other learned composition constituents appearing in both the video and language context, and their relationships. Extensive experiments validate the superior compositional generalizability of our approach, demonstrating its ability to handle queries with novel combinations of seen words as well as novel words in the testing composition.

Imagine This! Scripts to Compositions to Videos

COMPOSER: Compositional Reasoning of Group Activity in Videos with Keypoint-Only Modality

Compositional 3D-aware Video Generation with LLM Director

Crafting Parts for Expressive Object Composition

Learning Visual Composition through Improved Semantic Guidance

MotionCom: Automatic and Motion-Aware Image Composition with LLM and Video Diffusion Prior

DreamRunner: Fine-Grained Storytelling Video Generation with Retrieval-Augmented Motion Adaptation

PartCraft: Crafting Creative Objects by Parts

ICSVR: Investigating Compositional and Syntactic Understanding in Video Retrieval Models

Learning to Imagine: Visually-Augmented Natural Language Generation

Semantic Compositional Networks for Visual Captioning

CRAFT: Concept Recursive Activation FacTorization for Explainability

Summarize the Past to Predict the Future: Natural Language Descriptions of Context Boost Multimodal Object Interaction Anticipation

ComFusion: Enhancing Personalized Generation by Instance-Scene Compositing and Fusion

SceneCraft: An LLM Agent for Synthesizing 3D Scene as Blender Code

Intelligent Director: An Automatic Framework for Dynamic Visual Composition using ChatGPT

CRAFT: Cross-Attentional Flow Transformer for Robust Optical Flow

Enabling Visual Composition and Animation in Unsupervised Video Generation

RoboDreamer: Learning Compositional World Models for Robot Imagination

Variational Cross-Graph Reasoning and Adaptive Structured Semantics Learning for Compositional Temporal Grounding

COLA: A Benchmark for Compositional Text-to-image Retrieval