Abstract:In this paper, we investigate the problem of abductive visual reasoning (AVR), which requires vision systems to infer the most plausible explanation for visual observations. Unlike previous work which performs visual reasoning on static images or synthesized scenes, we exploit long-term reasoning from instructional videos that contain a wealth of detailed information about the physical world. We conceptualize two tasks for this emerging and challenging topic. The primary task is AVR, which is based on the initial configuration and desired goal from an instructional video, and the model is expected to figure out what is the most plausible sequence of steps to achieve the goal. In order to avoid trivial solutions based on appearance information rather than reasoning, the second task called AVR++ is constructed, which requires the model to answer why the unselected options are less plausible. We introduce a new dataset called VideoABC, which consists of 46,354 unique steps derived from 11,827 instructional videos, formulated as 13,526 abductive reasoning questions with an average reasoning duration of 51 seconds. Through an adversarial hard hypothesis mining algorithm, non-trivial and high-quality problems are generated efficiently and effectively. To achieve human-level reasoning, we propose a Hierarchical Dual Reasoning Network (HDRNet) to capture the long-term dependencies among steps and observations. We establish a benchmark for abductive visual reasoning, and our method set state-of-the-arts on AVR ( similar to 74 %) and AVR++ ( similar to 45 %), and humans can easily achieve over 90% accuracy on these two tasks. The large performance gap reveals the limitation of current video understanding models on temporal reasoning and leaves substantial room for future research on this challenging problem. Our dataset and code are available at https://github.com/wl-zhao/VideoABC .

Multi-modal Action Chain Abductive Reasoning

A Multi-Modal Context Reasoning Approach for Conditional Inference on Joint Textual and Visual Clues

Insight-V: Exploring Long-Chain Visual Reasoning with Multimodal Large Language Models

Complex Video Action Reasoning Via Learnable Markov Logic Network

Look, Remember and Reason: Grounded reasoning in videos with language models

Reasoning Grasping via Multimodal Large Language Model

Multimodal Chain-of-Thought Reasoning in Language Models

VideoABC: A Real-World Video Dataset for Abductive Visual Reasoning

Explainable Video Action Reasoning via Prior Knowledge and State Transitions

Language Model Guided Interpretable Video Action Reasoning

Multimodal Analogical Reasoning over Knowledge Graphs

MECD: Unlocking Multi-Event Causal Discovery in Video Reasoning

MaTCR: Modality-Aligned Thought Chain Reasoning for Multimodal Task-Oriented Dialogue Generation

Progressive Multimodal Reasoning via Active Retrieval

iReason: Multimodal Commonsense Reasoning using Videos and Natural Language with Interpretability

UNcommonsense Reasoning: Abductive Reasoning about Uncommon Situations

Language Models Can Improve Event Prediction by Few-Shot Abductive Reasoning

ReAct: Synergizing Reasoning and Acting in Language Models

The Curious Case of Nonverbal Abstract Reasoning with Multi-Modal Large Language Models

Enhancing Visual Reasoning with Autonomous Imagination in Multimodal Large Language Models