Abstract:The option framework, one of the most promising Hierarchical Reinforcement Learning (HRL) frameworks, is developed based on the Semi-Markov Decision Problem (SMDP) and employs a triple formulation of the option (i.e., an action policy, a termination probability, and an initiation set). These design choices, however, mean that the option framework: 1) has low sample efficiency, 2) cannot use more stable Markov Decision Problem (MDP) based learning algorithms, 3) represents abstract actions implicitly, and 4) is expensive to scale up. To overcome these problems, here we propose a simple yet effective MDP implementation of the option framework: the Skill-Action (SA) architecture. Derived from a novel discovery that the SMDP option framework has an MDP equivalence, SA hierarchically extracts skills (abstract actions) from primary actions and explicitly encodes these knowledge into skill context vectors (embedding vectors). Although SA is MDP formulated, skills can still be temporally extended by applying the attention mechanism to skill context vectors. Unlike the option framework, which requires M action policies for M skills, SA's action policy only needs one decoder to decode skill context vectors into primary actions. Under this formulation, SA can be optimized with any MDP based policy gradient algorithm. Moreover, it is sample efficient, cheap to scale up, and theoretically proven to have lower variance. Our empirical studies on challenging infinite horizon robot simulation environments demonstrate that SA not only outperforms all baselines by a large margin, but also exhibits smaller variance, faster convergence, and good interpretability. On transfer learning tasks, SA also outperforms the other models and shows its advantage on reusing knowledge across tasks. A potential impact of SA is to pave the way for a large scale pre-training architecture in the reinforcement learning area.

Adaptive Stage-Aware Assessment Skill Transfer for Skill Determination

Independent Skill Transfer for Deep Reinforcement Learning

Unsupervised Discovery of Transitional Skills for Deep Reinforcement Learning

GSC: A Graph-Based Skill Composition Framework for Robot Learning

Adaptive Action Assessment

Research on Automatic Assessment of Transferable Skills.

Automatic Modelling for Interactive Action Assessment

An Asymmetric Modeling for Action Assessment

SkillS: Adaptive Skill Sequencing for Efficient Temporally-Extended Exploration

Learning and fusing multiple hidden substages for action quality assessment

AdaptSim: Task-Driven Simulation Adaptation for Sim-to-Real Transfer

The Skill-Action Architecture: Learning Abstract Action Embeddings for Reinforcement Learning

Multi-Stage Decision-Making Skill Learning for Soccer Robot

Diff-Transfer: Model-based Robotic Manipulation Skill Transfer via Differentiable Physics Simulation

Skill Transfer Learning for Autonomous Robots and Human–robot Cooperation: A Survey

STAP: Sequencing Task-Agnostic Policies

ASC: Adaptive Skill Coordination for Robotic Mobile Manipulation

Attend, Adapt and Transfer: Attentive Deep Architecture for Adaptive Transfer from multiple sources in the same domain

Procedure-Aware Action Quality Assessment: Datasets and Performance Evaluation

Unsupervised Skill Discovery for Robotic Manipulation through Automatic Task Generation

Active Task Randomization: Learning Robust Skills via Unsupervised Generation of Diverse and Feasible Tasks