Abstract:The Credit Assignment Problem (CAP) refers to the longstanding challenge of Reinforcement Learning (RL) agents to associate actions with their long-term consequences. Solving the CAP is a crucial step towards the successful deployment of RL in the real world since most decision problems provide feedback that is noisy, delayed, and with little or no information about the causes. These conditions make it hard to distinguish serendipitous outcomes from those caused by informed decision-making. However, the mathematical nature of credit and the CAP remains poorly understood and defined. In this survey, we review the state of the art of Temporal Credit Assignment (CA) in deep RL. We propose a unifying formalism for credit that enables equitable comparisons of state-of-the-art algorithms and improves our understanding of the trade-offs between the various methods. We cast the CAP as the problem of learning the influence of an action over an outcome from a finite amount of experience. We discuss the challenges posed by delayed effects, transpositions, and a lack of action influence, and analyse how existing methods aim to address them. Finally, we survey the protocols to evaluate a credit assignment method and suggest ways to diagnose the sources of struggle for different methods. Overall, this survey provides an overview of the field for new-entry practitioners and researchers, it offers a coherent perspective for scholars looking to expedite the starting stages of a new study on the CAP, and it suggests potential directions for future research.

Selective Credit Assignment

Towards Practical Credit Assignment for Deep Reinforcement Learning

A Survey of Temporal Credit Assignment in Deep Reinforcement Learning

An Information-Theoretic Perspective on Credit Assignment in Reinforcement Learning

Would I have gotten that reward? Long-term credit assignment by counterfactual contribution analysis

Credit Assignment: Challenges and Opportunities in Developing Human-like AI Agents

Hindsight-DICE: Stable Credit Assignment for Deep Reinforcement Learning

Towards Causal Credit Assignment

Hindsight Network Credit Assignment: Efficient Credit Assignment in Networks of Discrete Stochastic Units

Neural mechanisms of credit assignment for delayed outcomes during contingent learning

Sequence Compression Speeds Up Credit Assignment in Reinforcement Learning

Learning to assign credit in reinforcement learning by incorporating abstract relations

Credit Assignment During Movement Reinforcement Learning.

Minimizing Control for Credit Assignment with Strong Feedback

On Credit Assignment in Hierarchical Reinforcement Learning

Temporal-Difference Learning Using Distributed Error Signals

Credit assignment with predictive contribution measurement in multi-agent reinforcement learning

Demystifying the Recency Heuristic in Temporal-Difference Learning

Attention-Based Deep Spiking Neural Networks for Temporal Credit Assignment Problems.

Asynchronous Credit Assignment Framework for Multi-Agent Reinforcement Learning

STCA: Spatio-Temporal Credit Assignment with Delayed Feedback in Deep Spiking Neural Networks