Abstract:Anticipatory thinking drives our ability to manage risk - identification and mitigation - in everyday life, from bringing an umbrella when it might rain to buying car insurance. As AI systems become part of everyday life, they too have begun to manage risk. Autonomous vehicles log millions of miles, StarCraft and Go agents have similar capabilities to humans, implicitly managing risks presented by their opponents. To further increase performance in these tasks, out-of-distribution evaluation can characterize a model's bias, what we view as a type of risk management. However, learning to identify and mitigate low-frequency, high-impact risks is at odds with the observational bias required to train machine learning models. StarCraft and Go are closed-world domains whose risks are known and mitigations well documented, ideal for learning through repetition. Adversarial filtering datasets provide difficult examples but are laborious to curate and static, both barriers to real-world risk management. Adversarial robustness focuses on model poisoning under the assumption there is an adversary with malicious intent, without considering naturally occurring adversarial examples. These methods are all important steps towards improving risk management but do so without considering open-worlds. We unify these open-world risk management challenges with two contributions. The first is our perception challenges, designed for agents with imperfect perceptions of their environment whose consequences have a high impact. Our second contribution are cognition challenges, designed for agents that must dynamically adjust their risk exposure as they identify new risks and learn new mitigations. Our goal with these challenges is to spur research into solutions that assess and improve the anticipatory thinking required by AI agents to manage risk in open-worlds and ultimately the real-world.

Anticipatory Thinking: A Testing and Representation Challenge for Self-Driving Cars

Anticipatory Thinking Challenges in Open Worlds: Risk Management

Intelligence Testing for Autonomous Vehicles: A New Approach.

Intelligent driving intelligence test for autonomous vehicles with naturalistic and adversarial environment

A Theoretical Foundation of Intelligence Testing and Its Application for Intelligent Vehicles

Ensuring Safe Autonomy: Navigating the Future of Autonomous Vehicles

Verisimilar Percept Sequences Tests for Autonomous Driving Intelligent Agent Assessment

Generating Edge Cases for Testing Autonomous Vehicles Using Real-World Data

Teaching Vehicles to Anticipate: A Systematic Study on Probabilistic Behavior Prediction Using Large Data Sets

Testing autonomous vehicles and AI: perspectives and challenges from cybersecurity, transparency, robustness and fairness

Disentangling Perception-failure-induced Corner Cases by Counterfactual Reasoning

The need of a self for self-driving cars a theoretical model applying homeostasis to self driving

Paving the Roadway for Safety of Automated Vehicles: An Empirical Study on Testing Challenges

Exiting the Simulation: The Road to Robust and Resilient Autonomous Vehicles at Scale

Towards Automated Safety Coverage and Testing for Autonomous Vehicles with Reinforcement Learning

Simulation-based Adversarial Test Generation for Autonomous Vehicles with Machine Learning Components

How to Guarantee Driving Safety for Autonomous Vehicles in a Real-World Environment: A Perspective on Self-Evolution Mechanisms

Adaptive Stress Testing for Autonomous Vehicles

Towards a Universal Evaluation Model for Careful and Competent Autonomous Driving

Adversarial Generation of Safety-Critical Lane-Change Scenarios for Autonomous Vehicles

Evolving Testing Scenario Generation Method and Intelligence Evaluation Framework for Automated Vehicles