Abstract:To date, transfer learning (TL) has been successfully applied for enhancing the learning performance of reinforcement learning (RL), and many transfer RL (TRL) approaches have been proposed in the literature. However, most of the existing TRL approaches consider knowledge transfer between RL tasks sharing the same state-action space. These methods thus may fail in cases where the RL tasks available for conducting knowledge transfer possess heterogeneous state-action spaces, which is common in many real-world applications. TRL across heterogeneous problem domains is challenging since the differences lie in the state-action spaces of the RL tasks are natural barriers in the knowledge transfer across tasks. This becomes more difficult if multiple heterogeneous source tasks are available when conducting knowledge transfer for a target RL task, as we have to identify the appropriate source task adaptively before performing knowledge transfer towards enhanced RL performance. In this article, we propose a new TRL algorithm with adaptive policy gradient transfer for the cases having multiple heterogeneous source RL tasks. The core ingredients of the proposed algorithm contain a source task selection module to select an appropriate task from a set of heterogeneous source tasks and a knowledge transfer module for conducting knowledge transfer across heterogeneous RL tasks. To investigate the performance of the proposed algorithm, we have conducted comprehensive empirical studies based on the well-known continuous robotic RL task with heterogeneous settings in the number of robot arms (links). The obtained results show that the proposed algorithm is effective and efficient in conducting knowledge transfer across heterogeneous problems for enhanced RL performance, over both the RL algorithm having no knowledge transfer in the learning process and the existing state-of-the-art TRL method.

Shaping in Reinforcement Learning Via Knowledge Transferred from Human-Demonstrations

Shaping in Reinforcement Learning by Knowledge Transferred from Human-Demonstrations of a Simple Similar Task.

Transferring knowledge from human-demonstration trajectories to reinforcement learning

DGTRL: Deep graph transfer reinforcement learning method based on fusion of knowledge and data

Shaping Progressive Net of Reinforcement Learning for Policy Transfer with Human Evaluative Feedback

Knowledge Sharing and Transfer via Centralized Reward Agent for Multi-Task Reinforcement Learning

Learning To Walk With Prior Knowledge

Deep Q-learning with Explainable and Transferable Domain Rules.

Mutual Information Based Knowledge Transfer Under State-Action Dimension Mismatch

Shaping Reward Learning Approach from Passive Samples

Hierarchical Reinforcement Learning from Demonstration via Reachability-Based Reward Shaping

Learning to Utilize Shaping Rewards: A New Approach of Reward Shaping

Reusing Source Task Knowledge Via Transfer Approximator in Reinforcement Transfer Learning

Shaping Rewards for Reinforcement Learning with Imperfect Demonstrations using Generative Models

Reinforcement Learning with Adaptive Policy Gradient Transfer Across Heterogeneous Problems

A new Potential-Based Reward Shaping for Reinforcement Learning Agent

Efficient Reinforcement Learning from Demonstration Via Bayesian Network-Based Knowledge Extraction

Efficient Deep Reinforcement Learning Through Policy Transfer.

ACQUISITION SLOPE SURFACE WALKING FOR HUMANOIDS VIA TRANSFER LEARNING

Combining Model-Based $Q$ -Learning with Structural Knowledge Transfer for Robot Skill Learning

Learning to Perform Trajectory Generation from Low-Quality Demonstrations