Abstract:Transfer learning in Reinforcement Learning (RL) has been widely studied to overcome training issues of Deep-RL, i.e., exploration cost, data availability and convergence time, by introducing a way to enhance training phase with external knowledge. Generally, knowledge is transferred from expert-agents to novices. While this fixes the issue for a novice agent, a good understanding of the task on expert agent is required for such transfer to be effective. As an alternative, in this paper we propose Expert-Free Online Transfer Learning (EF-OnTL), an algorithm that enables expert-free real-time dynamic transfer learning in multi-agent system. No dedicated expert exists, and transfer source agent and knowledge to be transferred are dynamically selected at each transfer step based on agents' performance and uncertainty. To improve uncertainty estimation, we also propose State Action Reward Next-State Random Network Distillation (sars-RND), an extension of RND that estimates uncertainty from RL agent-environment interaction. We demonstrate EF-OnTL effectiveness against a no-transfer scenario and advice-based baselines, with and without expert agents, in three benchmark tasks: Cart-Pole, a grid-based Multi-Team Predator-Prey (mt-pp) and Half Field Offense (HFO). Our results show that EF-OnTL achieve overall comparable performance when compared against advice-based baselines while not requiring any external input nor threshold tuning. EF-OnTL outperforms no-transfer with an improvement related to the complexity of the task addressed.

A Survey on Transfer Learning for Multiagent Reinforcement Learning Systems

Knowledge Transfer for Cross-Domain Reinforcement Learning: A Systematic Review

Transfer Learning in Deep Reinforcement Learning: A Survey

Multi-agent Reinforcement Learning: A Comprehensive Survey

KnowRU: Knowledge Reusing via Knowledge Distillation in Multi-agent Reinforcement Learning

A Survey and Critique of Multiagent Deep Reinforcement Learning

Multi-Agent Deep Reinforcement Learning for Multi-Robot Applications: A Survey

Parallel Knowledge Transfer in Multi-Agent Reinforcement Learning

KnowRU: Knowledge Reuse Via Knowledge Distillation in Multi-Agent Reinforcement Learning

Sim-to-Real Transfer in Deep Reinforcement Learning for Robotics: a Survey

Foundations for Transfer in Reinforcement Learning: A Taxonomy of Knowledge Modalities

Deep multiagent reinforcement learning: challenges and directions

A survey on multi-agent deep reinforcement learning: from the perspective of challenges and applications

Knowledge Reuse of Multi-Agent Reinforcement Learning in Cooperative Tasks

Enabling Inter-Agent Transfer for Multi-Agent Learning System by Incorporating Role Reversal

Improved Reinforcement Learning in Cooperative Multi-agent Environments Using Knowledge Transfer

Lateral Transfer Learning for Multiagent Reinforcement Learning.

Expert-Free Online Transfer Learning in Multi-Agent Reinforcement Learning

Multi-Agent Reinforcement Learning: A Report on Challenges and Approaches

Learning in Multi-Agent Systems with Sparse Interactions by Knowledge Transfer and Game Abstraction

Multi-Agent Reinforcement Learning for Autonomous Driving: A Survey