Abstract:Deep reinforcement learning is a powerful approach to complex decision making. However, one issue that limits its practical application is its brittleness, sometimes failing to train in the presence of small changes in the environment. This work is motivated by the empirical observation that directly applying an already trained model to a related task often works remarkably well, also called zero-shot transfer. We take this practical trick one step further to consider how to systematically select good tasks to train, maximizing overall performance across a range of tasks. Given the high cost of training, it is critical to choose a small set of training tasks. The key idea behind our approach is to explicitly model the performance loss (generalization gap) incurred by transferring a trained model. We hence introduce Model-Based Transfer Learning (MBTL) for solving contextual RL problems. In this work, we model the performance loss as a simple linear function of task context similarity. Furthermore, we leverage Bayesian optimization techniques to efficiently model and estimate the unknown training performance of the task space. We theoretically show that the method exhibits regret that is sublinear in the number of training tasks and discuss conditions to further tighten regret bounds. We experimentally validate our methods using urban traffic and standard control benchmarks. Despite the conceptual simplicity, the experimental results suggest that MBTL can achieve greater performance than strong baselines, including exhaustive training on all tasks, multi-task training, and random selection of training tasks. This work lays the foundations for investigating explicit modeling of generalization, thereby enabling principled yet effective methods for contextual RL.

Model-Based Transfer Reinforcement Learning Based on Graphical Model Representations

Self-Supervised Reinforcement Learning that Transfers using Random Features

Shaping in Reinforcement Learning Via Knowledge Transferred from Human-Demonstrations

Explainable Reinforcement Learning via Model Transforms

Deep Reinforcement Learning for Autonomous Driving by Transferring Visual Features.

A Survey on Explainable Reinforcement Learning: Concepts, Algorithms, Challenges

A Reliable Representation with Bidirectional Transition Model for Visual Reinforcement Learning Generalization

Theory-based Causal Transfer: Integrating Instance-level Induction and Abstract-level Structure Learning

DGTRL: Deep graph transfer reinforcement learning method based on fusion of knowledge and data

Deep Q-learning with Explainable and Transferable Domain Rules.

Bridging Scenarios in Reinforcement Learning with Continuously Generated Relaying Predictive Models.

Reason more like human: Incorporating meta information into hierarchical reinforcement learning for knowledge graph reasoning

Multi-hop Knowledge Reasoning with Deep Reinforcement Learning

Explanation of Reinforcement Learning Model in Dynamic Multi-Agent System

Foundations for Transfer in Reinforcement Learning: A Taxonomy of Knowledge Modalities

Grounding Language for Transfer in Deep Reinforcement Learning

Data-Efficient Model-Based Reinforcement Learning For Robot Control

Attention-based Exploitation and Exploration Strategy for Multi-Hop Knowledge Graph Reasoning

Explainable Reinforcement Learning via a Causal World Model

Model-Based Transfer Learning for Contextual Reinforcement Learning

Towards Generalizable Reinforcement Learning via Causality-Guided Self-Adaptive Representations