Abstract:In this paper, we consider a transfer reinforcement learning problem involving agents with different action spaces. Specifically, for any new unseen task, the goal is to use a successful demonstration of this task by an expert agent in its action space to enable a learner agent learn an optimal policy in its own different action space with fewer samples than those required if the learner was learning on its own. Existing transfer learning methods across different action spaces either require handcrafted mappings between those action spaces provided by human experts, which can induce bias in the learning procedure, or require the expert agent to share its policy parameters with the learner agent, which does not generalize well to unseen tasks. In this work, we propose a method that learns a subgoal mapping between the expert agent policy and the learner agent policy. Since the expert agent and the learner agent have different action spaces, their optimal policies can have different subgoal trajectories. We learn this subgoal mapping by training a Long Short Term Memory (LSTM) network for a distribution of tasks and then use this mapping to predict the learner subgoal sequence for unseen tasks, thereby improving the speed of learning by biasing the agent's policy towards the predicted learner subgoal sequence. Through numerical experiments, we demonstrate that the proposed learning scheme can effectively find the subgoal mapping underlying the given distribution of tasks. Moreover, letting the learner agent imitate the expert agent's policy with the learnt subgoal mapping can significantly improve the sample efficiency and training time of the learner agent in unseen new tasks.

TranSlider: Transfer Ensemble Learning from Exploitation to Exploration

Unsupervised Discovery of Transitional Skills for Deep Reinforcement Learning

Ensemble Transfer Learning Algorithm

TRS: Transferability Reduced Ensemble Via Promoting Gradient Diversity and Model Smoothness

Transferring knowledge from human-demonstration trajectories to reinforcement learning

Shaping in Reinforcement Learning Via Knowledge Transferred from Human-Demonstrations

Meta-Transfer Networks for Zero-Shot Learning

Deep Model Transferability from Attribution Maps

Ensemble Transfer Strategy Based on Domain Difference for Robot Multiple Peg-in-Hole Assembly

Transfer learning for ensembles: reducing computation time and keeping the diversity

Overcoming the Sim-to-Real Gap: Leveraging Simulation to Learn to Explore for Real-World RL

Transfer Reinforcement Learning in Heterogeneous Action Spaces using Subgoal Mapping

Is Exploration All You Need? Effective Exploration Characteristics for Transfer in Reinforcement Learning

Solution Transfer in Evolutionary Optimization: An Empirical Study on Sequential Transfer

Ensemble Diversity Facilitates Adversarial Transferability

Learning What and Where to Transfer

Scalable Transfer Learning with Expert Models

Transfer Regression with Data-Augmented Ensemble Learning Framework

TransAgent: Transfer Vision-Language Foundation Models with Heterogeneous Agent Collaboration

Boosting the Transferability of Ensemble Adversarial Attack via Stochastic Average Variance Descent

Introspective Action Advising for Interpretable Transfer Learning