Abstract:Humans have a remarkable ability to make decisions by accurately reasoning about future events, including the future behaviors and states of mind of other agents. Consider driving a car through a busy intersection: it is necessary to reason about the physics of the vehicle, the intentions of other drivers, and their beliefs about your own intentions. If you signal a turn, another driver might yield to you, or if you enter the passing lane, another driver might decelerate to give you room to merge in front. Competent drivers must plan how they can safely react to a variety of potential future behaviors of other agents before they make their next move. This requires contingency planning: explicitly planning a set of conditional actions that depend on the stochastic outcome of future events. In this work, we develop a general-purpose contingency planner that is learned end-to-end using high-dimensional scene observations and low-dimensional behavioral observations. We use a conditional autoregressive flow model to create a compact contingency planning space, and show how this model can tractably learn contingencies from behavioral observations. We developed a closed-loop control benchmark of realistic multi-agent scenarios in a driving simulator (CARLA), on which we compare our method to various noncontingent methods that reason about multi-agent future behavior, including several state-of-the-art deep learning-based planning approaches. We illustrate that these noncontingent planning methods fundamentally fail on this benchmark, and find that our deep contingency planning method achieves significantly superior performance. Code to run our benchmark and reproduce our results is available at <a class="link-external link-https" href="https://sites.google.com/view/contingency-planning" rel="external noopener nofollow">this https URL</a>

Investigating Contingency Awareness Using Atari 2600 Games

Contingency-Aware Exploration in Reinforcement Learning

Contingency Games for Multi-Agent Interaction

Probing Transfer in Deep Reinforcement Learning without Task Engineering

Contingencies from Observations: Tractable Contingency Planning with Learned Behavior Models

Playing Atari Games with Deep Reinforcement Learning and Human Checkpoint Replay

HackAtari: Atari Learning Environments for Robust and Continual Reinforcement Learning

Unifying Count-Based Exploration and Intrinsic Motivation

Virtual Augmented Reality for Atari Reinforcement Learning

Analysis of Agent Expertise in Ms. Pac-Man using Value-of-Information-based Policies

Human-level control through deep reinforcement learning

Efficient and Scalable Exploration Via Estimation-Error

Shielding Atari Games with Bounded Prescience

Detecting contingencies: an infomax approach

Playing Atari with Deep Reinforcement Learning

Learn to Interpret Atari Agents.

Towards a Standardized Reinforcement Learning Framework for AAM Contingency Management

Approximate Shielding of Atari Agents for Safe Exploration

Interactive Visual Reasoning under Uncertainty

Model-Based Reinforcement Learning for Atari