Abstract:Safe reinforcement learning deals with mitigating or avoiding unsafe situations by reinforcement learning (RL) agents. Safe RL approaches are based on specific risk representations for particular problems or domains. In order to analyze agent behaviors, compare safe RL approaches, and effectively transfer techniques between application domains, it is necessary to understand the types of risk specific to safe RL problems. We performed a systematic literature mapping with the objective to characterize risk in safe RL. Based on the obtained results, we present definitions, characteristics, and types of risk that hold on multiple application domains. Our literature mapping covers literature from the last 5 years (2017-2022), from a variety of knowledge areas (AI, finance, engineering, medicine) where RL approaches emphasize risk representation and management. Our mapping covers 72 papers filtered systematically from over thousands of papers on the topic. Our proposed notion of risk covers a variety of representations, disciplinary differences, common training exercises, and types of techniques. We encourage researchers to include explicit and detailed accounts of risk in future safe RL research reports, using this mapping as a starting point. With this information, researchers and practitioners could draw stronger conclusions on the effectiveness of techniques on different problems.

What problem does this paper attempt to address?

The problem that this paper attempts to solve is how to define and represent risks in Reinforcement Learning (RL). Specifically, the paper focuses on how to identify, describe, and classify safety - related risk types in different application domains to promote technology transfer and comparative research between different domains. Through a systematic literature mapping study, the authors aim to provide a cross - domain risk definition, as well as a series of attributes and values that can be used to represent risk factors in any application domain. ### Main contributions of the paper: 1. **Proposing the definition of risk**: The paper proposes a risk definition applicable to the current literature and provides a method to identify, describe, and label the risk considerations faced by safety - reinforced learning agents. 2. **Attributes and values of risk factors**: The paper defines the attributes and values that risk factors may have, regardless of the application domain. These attributes include Source, Target, Severity, Uncertainty, Interference, Frequency, and Timing. 3. **Common problems and risk representation**: The paper studies the common problem types and risk representations in safety - reinforced learning, providing references for future research. ### Research background: - **Reinforcement learning**: Reinforcement learning is a method of learning optimal strategies through interaction with the environment, but this learning process may place agents in dangerous or unsafe environments. - **Safety - reinforced learning**: To reduce or avoid the occurrence of unsafe situations, safety - reinforced learning extends traditional reinforcement learning by explicitly modeling the safety of agents and their actions. ### Research methods: - **Systematic literature mapping**: The paper adopts the systematic literature mapping method and screens out 72 relevant papers from the literature from 2017 to 2022 for detailed analysis. - **Research questions**: The paper focuses on three main research questions: - RQ1: What are the characteristics of risks in reinforcement learning literature? - RQ2: Which application domains explicitly model risks in reinforcement learning? - RQ3: How to represent risks in reinforcement learning? ### Main findings: - **Multi - dimensional representation of risks**: The paper emphasizes that the representation of risks is not only about loss, variance, or randomness, but also includes multiple aspects such as risk type, target, source, severity, frequency, and timing. - **Cross - domain technology transfer**: Through systematic literature analysis, the paper finds that there are similar risk types between different domains, which provides the possibility for cross - domain technology transfer. - **Future research directions**: The paper suggests that researchers should include clear and detailed descriptions of risks in future research reports to promote more effective technology evaluation and application. ### Conclusion: Through the systematic analysis of a large number of literatures, the paper provides a comprehensive risk definition and representation framework for the field of safety - reinforced learning. This framework not only helps to understand the risk characteristics in different application domains, but also provides an important reference for future research and technology transfer.

Modeling Risk in Reinforcement Learning: A Literature Mapping

Cautious Adaptation For Reinforcement Learning in Safety-Critical Settings

Risk-Sensitive and Robust Model-Based Reinforcement Learning and Planning

ROSCOM: Robust Safe Reinforcement Learning on Stochastic Constraint Manifolds

Safe and Robust Reinforcement Learning: Principles and Practice

One Risk to Rule Them All: A Risk-Sensitive Perspective on Model-Based Offline Reinforcement Learning

Look Before You Leap: Safe Model-Based Reinforcement Learning with Human Intervention

Modeling and Interpreting Real-world Human Risk Decision Making with Inverse Reinforcement Learning

Train Trajectory Optimization with High-Risk State Space Boundaries: A Safe Reinforcement Learning Approach

Safe Model-Based Reinforcement Learning with an Uncertainty-Aware Reachability Certificate

State-wise Safe Reinforcement Learning: A Survey

Evaluating Model-free Reinforcement Learning Toward Safety-critical Tasks

A Review of Safe Reinforcement Learning: Methods, Theories, and Applications

Risk-Averse Model Uncertainty for Distributionally Robust Safe Reinforcement Learning

How to Learn from Risk: Explicit Risk-Utility Reinforcement Learning for Efficient and Safe Driving Strategies

Risk-Aware Reinforcement Learning through Optimal Transport Theory

A Review of Safe Reinforcement Learning: Methods, Theory and Applications

Safeguarded Progress in Reinforcement Learning: Safe Bayesian Exploration for Control Policy Synthesis

Learning to be Safe: Deep RL with a Safety Critic

Lyapunov-based uncertainty-aware safe reinforcement learning

Iterative Reachability Estimation for Safe Reinforcement Learning