Shared-unique Features and Task-aware Prioritized Sampling on Multi-task Reinforcement Learning

Po-Shao Lin,Jia-Fong Yeh,Yi-Ting Chen,Winston H. Hsu

2024-06-02

Abstract:We observe that current state-of-the-art (SOTA) methods suffer from the performance imbalance issue when performing multi-task reinforcement learning (MTRL) tasks. While these methods may achieve impressive performance on average, they perform extremely poorly on a few tasks. To address this, we propose a new and effective method called STARS, which consists of two novel strategies: a shared-unique feature extractor and task-aware prioritized sampling. First, the shared-unique feature extractor learns both shared and task-specific features to enable better synergy of knowledge between different tasks. Second, the task-aware sampling strategy is combined with the prioritized experience replay for efficient learning on tasks with poor performance. The effectiveness and stability of our STARS are verified through experiments on the mainstream Meta-World benchmark. From the results, our STARS statistically outperforms current SOTA methods and alleviates the performance imbalance issue. Besides, we visualize the learned features to support our claims and enhance the interpretability of STARS.

Machine Learning,Artificial Intelligence

What problem does this paper attempt to address?

This paper focuses on the issue of performance imbalance in Multi-Task Reinforcement Learning (MTRL). Existing state-of-the-art methods may perform well on average, but poorly on specific tasks. To address this problem, the paper proposes a new approach called STARS, which consists of two innovative strategies: Shared-Unique Feature Extractor and Task-Aware Priority Sampling. The Shared-Unique Feature Extractor learns shared features among tasks and task-specific features to facilitate knowledge collaboration across different tasks. The Task-Aware Sampling Strategy combines prioritized experience replay to efficiently handle underperforming tasks and dynamically adjust the number of samples. Experiments on the mainstream Meta-World benchmark validate STARS, showing that it statistically outperforms the current state-of-the-art methods and mitigates the performance imbalance issue. Additionally, the paper enhances the interpretability of STARS by visualizing the learned features. The paper suggests that the performance imbalance issue may arise from the ineffective utilization of shared and unique task features, as well as the lack of dynamic attention adjustment based on task performance differences. STARS addresses these issues through its design, improving stability and performance across tasks.

Shared-unique Features and Task-aware Prioritized Sampling on Multi-task Reinforcement Learning

T3S: Improving Multi-Task Reinforcement Learning with Task-Specific Feature Selector and Scheduler.

Efficient Multi-Task Reinforcement Learning via Task-Specific Action Correction

Prioritized Hindsight with Dual Buffer for Meta-Reinforcement Learning

On Steering Multi-Annotations per Sample for Multi-Task Learning

Leveraging the Efficiency of Multi-Task Robot Manipulation Via Task-Evoked Planner and Reinforcement Learning

Multi-Task Reinforcement Learning with Attention-Based Mixture of Experts.

Sample Efficient Myopic Exploration Through Multitask Reinforcement Learning with Diverse Tasks

Sample-efficient multi-agent reinforcement learning with masked reconstruction

Meta Reinforcement Learning with Task Embedding and Shared Policy.

Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning

Selective Learning for Sample-Efficient Training in Multi-Agent Sparse Reward Tasks

Continuous Self-Adaptive Optimization to Learn Multi-Task Multi-Agent

Multi-Task Reinforcement Learning with Mixture of Orthogonal Experts

A Brain-Inspired Incremental Multi-task Reinforcement Learning Approach

AdaTask: A Task-aware Adaptive Learning Rate Approach to Multi-task Learning

STAS: Spatial-Temporal Return Decomposition for Solving Sparse Rewards Problems in Multi-agent Reinforcement Learning

Sharing Knowledge in Multi-Task Deep Reinforcement Learning

ST-MAML: A Stochastic-Task based Method for Task-Heterogeneous Meta-Learning

A Multi-Task Approach to Robust Deep Reinforcement Learning for Resource Allocation

Hierarchical Attention Master–Slave for Heterogeneous Multi-Agent Reinforcement Learning