Feudal Graph Reinforcement Learning

Tommaso Marzi,Arshjot Khehra,Andrea Cini,Cesare Alippi

2024-10-14

Abstract:Graph-based representations and message-passing modular policies constitute prominent approaches to tackling composable control problems in reinforcement learning (RL). However, as shown by recent graph deep learning literature, such local message-passing operators can create information bottlenecks and hinder global coordination. The issue becomes more serious in tasks requiring high-level planning. In this work, we propose a novel methodology, named Feudal Graph Reinforcement Learning (FGRL), that addresses such challenges by relying on hierarchical RL and a pyramidal message-passing architecture. In particular, FGRL defines a hierarchy of policies where high-level commands are propagated from the top of the hierarchy down through a layered graph structure. The bottom layers mimic the morphology of the physical system, while the upper layers correspond to higher-order sub-modules. The resulting agents are then characterized by a committee of policies where actions at a certain level set goals for the level below, thus implementing a hierarchical decision-making structure that can naturally implement task decomposition. We evaluate the proposed framework on a graph clustering problem and MuJoCo locomotion tasks; simulation results show that FGRL compares favorably against relevant baselines. Furthermore, an in-depth analysis of the command propagation mechanism provides evidence that the introduced message-passing scheme favors learning hierarchical decision-making policies.

Machine Learning

What problem does this paper attempt to address?

The problem that this paper attempts to solve is the challenge of achieving high - level coordination in a modular architecture. Specifically, existing graph - based message - passing strategies may encounter information bottlenecks when dealing with tasks requiring high - level planning, which hinders global coordination. These problems are particularly prominent in complex systems (such as robots) that require distributed decision - making and control. The paper proposes a new method - Feudal Graph Reinforcement Learning (FGRL), which addresses these challenges by introducing hierarchical reinforcement learning and a pyramidal message - passing architecture. FGRL designs a multi - layer graph structure, where each node corresponds to a part of the system, high - level nodes are responsible for making high - level decisions, and low - level nodes execute specific control tasks. This method aims to promote effective information transfer and coordination between different levels, thereby improving the overall performance of the system.

The main contributions of the paper include:
1. Introducing the FGRL paradigm, which is a new deep - learning framework for graph - based hierarchical reinforcement learning in a composable environment.
2. Evaluating the implementation of this method on graph - clustering problems and continuous - control tasks in the MuJoCo motion benchmark, and the results show that FGRL is competitive compared to other baseline methods for composable control.
3. Providing experimental evidence to support the adoption of a hierarchical message - passing scheme and a graph - based representation to implement a hierarchical decision - making strategy.

Through these contributions, the paper opens up new avenues for designing deep RL architectures that incorporate biases aligned with the HRL agent structure.