Abstract:Using Causal Induction in Humans to Learn and Infer Causality from Video Amy Fire (amy.fire@ucla.edu) Song-Chun Zhu (sczhu@stat.ucla.edu) Center for Vision, Cognition, Learning, and Art University of California, Los Angeles Los Angeles, CA 90095 USA Abstract For both human and machine learners, it is a challenge to make high-level sense of observations by identifying causes, effects, and their connections. Once these connections are learned, the knowledge can be used to infer causes and effects where visual data might be partially hidden or ambiguous. In this paper, we present a Bayesian grammar model for human-perceived causal relationships that is learnable from video. Two exper- iments investigate high-level causal induction from low-level visual cues. In the first experiment, we show that a computer can apply known heuristics used for causal induction by hu- mans to learn perceptual causal relationships. In the second experiment, we show that our learned model can represent hu- mans’ performance in reasoning about hidden effects in video, even when the computer initially misdetects those effects. Keywords: Perceptual causality; causal induction; statistical models. Introduction A man approaches a closed door. He reaches out to grasp the handle and then stands there. Is it locked? Does he not have the key? He knocks and waits, but the door remains closed. Is there no one on the other side to open it? Watching these events unfold, humans can readily answer these questions based on their causal knowledge. One way humans can learn causal relationships is through daily ob- servation by internally measuring co-occurrence of events (Griffiths & Tenenbaum, 2005). Research suggests that humans use a few heuristics to determine whether a co- occurrence is causal, including: • whether the temporal lag between cause and effect is short, and the cause precedes the effect (Carey, 2009) and • whether agent actions are responsible for causes (Saxe, Tenenbaum, & Carey, 2005). However, learning from daily observation is limited: many actions and effects are hidden. Our prior knowledge about causal relationships between actions and effects allows us to fill in information about the events in the scene. Some current models represent knowledge with Bayesian networks, e.g., (Griffiths & Tenenbaum, 2005). These mod- els, however, are disjoint from the low-level visual data that people observe. Instead, models are built using high-level an- notations. In reality, agents build knowledge by observing low-level visual data, and models need to be able to deal with uncertainty in observation. Although Bayesian networks are commonly used to repre- sent causality (Pearl, 2009), grammar models have the ex- pressive power to represent a greater breadth of possibili- ties than a single instance of a Bayesian network (Griffiths & Tenenbaum, 2007). Grammar models allow for multiple configurations and high-level structures, making them more suitable for applications grounded on visual cues; Bayesian networks lack the representative power needed for this. Grammar models are represented graphically in the And- Or Graph (AOG). In the AOG, Or-nodes represent the mul- tiple alternatives, and And-nodes represent hierarchical de- compositions. The AOG naturally lends itself to represent causation where multiple alternative causes can lead to an ef- fect, and each cause is composed of conditions necessary for the effect. In this paper, we introduce a grammar model for repre- senting causal relationships between actions and object-status changes, the Causal And-Or Graph (C-AOG). We describe methods for learning the model by using co-occurrence to identify potential causal relationships between events and ap- plying the heuristics listed above to those potential relation- ships. In two experiments, we investigate how the model matches human perceptions of causality. Experiment 1 uses input typical of computer vision detection systems to investi- gate learning the C-AOG and human perceptions of causality. Experiment 2 demonstrates that the C-AOG models human judgments on imputing hidden variables from video. A Grammar Model for Causality In this section, we introduce the Causal And-Or Graph for causal reasoning, which ties agent actions to fluents. Fluents and Actions Specifically defining those object statuses that vary over time, the term fluents comes from the commonsense-reasoning lit- erature (Mueller, 2006). Relevant here are two kinds of flu- ents that intentional agents can change: object fluents (e.g., a light can be on or off) and fluents of the mind (e.g., an agent can be thirsty or not thirsty). The values of these fluents change as a result of agent ac- tions and also trigger rational agents to take action. A lack of change-inducing action (also known as the inertial action) causes the fluent to maintain its value; for example, a door that is closed will remain closed until some action changes it. In this work, fluents are modeled discriminatively. Actions (A i ) are modeled using the Temporal And-Or Graph (T-AOG), a grammar model for actions (Pei, Jia, & Zhu, 2011). In the T-AOG, And-nodes group the necessary ways for an action to be performed that allow detection of the action (e.g., object/agent spatial relations, agent poses, scene contexts, and temporal relationships), and Or-nodes provide

Actional-Perceptual Causality: Concepts and Inductive Learning for AI and Robotics

Learning perceptual causality from video

Causal Induction from Visual Observations for Goal Directed Tasks

From Continual Learning to Causal Discovery in Robotics

Deep causal learning for robotic intelligence

Towards a Grounded Theory of Causation for Embodied AI

From Semantics to Execution: Integrating Action Planning With Reinforcement Learning for Robotic Causal Problem-Solving

Learning Low-Level Causal Relations using a Simulated Robotic Arm

Using Causal Induction in Humans to Learn and Infer Causality from Video.

Causal Reinforcement Learning: A Survey

Causal Curiosity: RL Agents Discovering Self-supervised Experiments for Causal Representation Learning

Causal reasoning in typical computer vision tasks

Active Causal Structure Learning with Latent Variables: Towards Learning to Detour in Autonomous Robots

Causal-Based Approaches to Explain and Learn from Self-Extension—A Review

Learning by Doing: An Online Causal Reinforcement Learning Framework with Causal-Aware Policy

Causal Action Influence Aware Counterfactual Data Augmentation

Causal Reinforcement Learning for Optimisation of Robot Dynamics in Unknown Environments

Towards Understanding How Machines Can Learn Causal Overhypotheses

Relation-First Modeling Paradigm for Causal Representation Learning Toward the Development of AGI

Learning and inferring causality from video