Abstract:Abstract In computer science, reinforcement learning is a powerful framework with which artificial agents can learn to maximize their performance for any given Markov decision process (MDP). Advances over the last decade, in combination with deep neural networks, have enjoyed performance advantages over humans in many difficult task settings. However, such frameworks perform far less favorably when evaluated in their ability to generalize or transfer representations across different tasks. Existing algorithms that facilitate transfer typically are limited to cases in which the transition function or the optimal policy is portable to new contexts, but achieving “deep transfer” characteristic of human behavior has been elusive. Such transfer typically requires discovery of abstractions that permit analogical reuse of previously learned representations to superficially distinct tasks. Here, we demonstrate that abstractions that minimize error in predictions of reward outcomes generalize across tasks with different transition and reward functions. Such reward-predictive representations compress the state space of a task into a lower dimensional representation by combining states that are equivalent in terms of both the transition and reward functions. Because only state equivalences are considered, the resulting state representation is not tied to the transition and reward functions themselves and thus generalizes across tasks with different reward and transition functions. These results contrast with those using abstractions that myopically maximize reward in any given MDP and motivate further experiments in humans and animals to investigate if neural and cognitive systems involved in state representation perform abstractions that facilitate such equivalence relations. Author summary Humans are capable of transferring abstract knowledge from one task to another. For example, in a right-hand-drive country, a driver has to use the right arm to operate the shifter. A driver who learned how to drive in a right-hand-drive country can adapt to operating a left-hand-drive car and use the other arm for shifting instead of re-learning how to drive. Despite the fact that both tasks require different coordination of motor skills, both tasks are the same in an abstract sense: In both tasks, a car is operated and there is the same progression from 1st to 2nd gear and so on. We study distinct algorithms by which a reinforcement learning agent can discover state representations that encode knowledge about a particular task, and evaluate how well they can generalize. Through a sequence of simulation results, we show that state abstractions that minimize errors in prediction about future reward outcomes generalize across tasks, even those that superficially differ in both the goals (rewards) and the transitions from one state to the next. This work motivates biological studies to determine if distinct circuits are adapted to maximize reward vs. to discover useful state representations.

Generalizing Across Multi-Objective Reward Functions in Deep Reinforcement Learning

Reward-predictive representations generalize across tasks in reinforcement learning

Inverse Reinforcement Learning with Multiple Ranked Experts

Unbiased Methods for Multi-Goal Reinforcement Learning

Multi-objective reward generalization: improving performance of Deep Reinforcement Learning for applications in single-asset trading

A Survey Analyzing Generalization in Deep Reinforcement Learning

Effective Reward Specification in Deep Reinforcement Learning

A Generalized Acquisition Function for Preference-based Reward Learning

Improvements on Hindsight Learning

Unlock the Cognitive Generalization of Deep Reinforcement Learning Via Granular Ball Representation

Rethinking Information Structures in RLHF: Reward Generalization from a Graph Theory Perspective

Learning Long-Term Reward Redistribution via Randomized Return Decomposition

A Two-Stage Multi-Objective Deep Reinforcement Learning Framework.

Addressing Reward Engineering For Deep Reinforcement Learning On Multi-Stage Task

Predicting optimal value functions by interpolating reward functions in scalarized multi-objective reinforcement learning

Enhanced Generalization through Prioritization and Diversity in Self-Imitation Reinforcement Learning over Procedural Environments with Sparse Rewards

Reward Generalization in RLHF: A Topological Perspective

A Game-Theoretic Perspective of Generalization in Reinforcement Learning

Efficient Hindsight Reinforcement Learning Using Demonstrations for Robotic Tasks with Sparse Rewards

Large Language Models as Efficient Reward Function Searchers for Custom-Environment Multi-Objective Reinforcement Learning

PMDRL: Pareto-front-based Multi-Objective Deep Reinforcement Learning