Abstract:We often specify tasks for a robot using temporal language that can include different levels of abstraction. For example, the command “go to the kitchen before going to the second floor” contains spatial abstraction, given that “floor” consists of individual rooms that can also be referred to in isolation (“kitchen”, for example). There is also a temporal ordering of events, defined by the word “before”. Previous works have used syntactically co-safe Linear Temporal Logic (sc-LTL) to interpret temporal language (such as “before”), and Abstract Markov Decision Processes (AMDPs) to interpret hierarchical abstractions (such as “kitchen” and “second floor”), separately. To handle both types of commands at once, we introduce the Abstract Product Markov Decision Process (AP-MDP), a novel approach capable of representing non-Markovian reward functions at different levels of abstractions. The AP-MDP framework translates LTL into its corresponding automata, creates a product Markov Decision Process (MDP) of the LTL specification and the environment MDP, and decomposes the problem into subproblems to enable efficient planning with abstractions. AP-MDP performs faster than a non-hierarchical method of solving LTL problems in over 95 % of path planning tasks, and this number only increases as the size of the environment domain increases. In a cleanup world domain, AP-MDP performs faster in over 98% of tasks. We also present a neural sequence-to-sequence model trained to translate language commands into LTL expression, and a new corpus of non-Markovian language commands spanning different levels of abstraction. We test our framework with the collected language commands on two drones, demonstrating that our approach enables robots to efficiently solve temporal commands at different levels of abstraction in both indoor and outdoor environments.

Hierarchy through Composition with Linearly Solvable Markov Decision Processes

Hierarchical Average-Reward Linearly-solvable Markov Decision Processes

Compositional Planning for Logically Constrained Multi-Agent Markov Decision Processes

Compositional planning in Markov decision processes: Temporal abstraction meets generalized logic composition

Hierarchical Subtask Discovery With Non-Negative Matrix Factorization

Parallel Automatic Hierarchy in Hierarchical Reinforcement Learning

Hierarchical Method for Cooperative Multiagent Reinforcement Learning in Markov Decision Processes

Automatic formation of the structure of abstract machines in hierarchical reinforcement learning with state clustering

Guiding Multi-agent Multi-task Reinforcement Learning by a Hierarchical Framework with Logical Reward Shaping

Hybrid Recurrent Models Support Emergent Descriptions for Hierarchical Planning and Control

Hierarchical Planning with State Abstractions for Temporal Task Specifications.

A Hierarchical Framework for Cooperative Tasks in Multi-agent Systems

LLM Augmented Hierarchical Agents

Abstract Value Iteration for Hierarchical Reinforcement Learning

Deep Hierarchical Reinforcement Learning Algorithm in Partially Observable Markov Decision Processes

Compositional Foundation Models for Hierarchical Planning

Exploring the limits of Hierarchical World Models in Reinforcement Learning

Hierarchical Reinforcement Learning for Concurrent Discovery of Compound and Composable Policies

Multi-Horizon Representations with Hierarchical Forward Models for Reinforcement Learning

Learning for Decentralized Control of Multiagent Systems in Large, Partially-Observable Stochastic Environments

Hierarchical Reinforcement Learning with Opponent Modeling for Distributed Multi-agent Cooperation