Abstract:Sample efficiency is central to developing practical reinforcement learning (RL) for complex and large-scale decision-making problems. The ability to transfer and generalize knowledge gained from previous experiences to downstream tasks can significantly improve sample efficiency. Recent research indicates that successor feature (SF) RL algorithms enable knowledge generalization between tasks with different rewards but identical transition dynamics. It has recently been hypothesized that combining model-based (MB) methods with SF algorithms can alleviate the limitation of fixed transition dynamics. Furthermore, uncertainty-aware exploration is widely recognized as another appealing approach for improving sample efficiency. Putting together two ideas of hybrid model-based successor feature (MB-SF) and uncertainty leads to an approach to the problem of sample efficient uncertainty-aware knowledge transfer across tasks with different transition dynamics or/and reward functions. In this paper, the uncertainty of the value of each action is approximated by a Kalman filter (KF)-based multiple-model adaptive estimation. This KF-based framework treats the parameters of a model as random variables. To the best of our knowledge, this is the first attempt at formulating a hybrid MB-SF algorithm capable of generalizing knowledge across large or continuous state space tasks with various transition dynamics while requiring less computation at decision time than MB methods. The number of samples required to learn the tasks was compared to recent SF and MB baselines. The results show that our algorithm generalizes its knowledge across different transition dynamics, learns downstream tasks with significantly fewer samples than starting from scratch, and outperforms existing approaches.

Optimistic Linear Support and Successor Features as a Basis for Optimal Policy Transfer

Successive Convex Approximation Based Off-Policy Optimization for Constrained Reinforcement Learning

Transfer in Deep Reinforcement Learning Using Successor Features and Generalised Policy Improvement

Planning with a Learned Policy Basis to Optimally Solve Complex Tasks

Safety-Constrained Policy Transfer with Successor Features

Advantages and Limitations of using Successor Features for Transfer in Reinforcement Learning

An advantage based policy transfer algorithm for reinforcement learning with measures of transferability

Efficient Deep Reinforcement Learning Via Policy-Extended Successor Feature Approximator

SF-DQN: Provable Knowledge Transfer using Successor Feature for Deep Reinforcement Learning

Efficient Exploration for Multi-Agent Reinforcement Learning Via Transferable Successor Features

Policy Learning for Off-Dynamics RL with Deficient Support

Combining Behaviors with the Successor Features Keyboard

Combinatorial Optimization with Policy Adaptation using Latent Space Search

Imagined Value Gradients: Model-Based Policy Optimization with Transferable Latent Dynamics Models

Decision-Focused Model-based Reinforcement Learning for Reward Transfer

Efficient Deep Reinforcement Learning Via Adaptive Policy Transfer

Efficient Deep Reinforcement Learning Through Policy Transfer.

Uncertainty-aware transfer across tasks using hybrid model-based successor feature reinforcement learning

Learning Successor Features the Simple Way

Non-Adversarial Inverse Reinforcement Learning via Successor Feature Matching

IOB: Integrating Optimization Transfer and Behavior Transfer for Multi-Policy Reuse