Towards Human Awareness in Robot Task Planning with Large Language Models

Yuchen Liu,Luigi Palmieri,Sebastian Koch,Ilche Georgievski,Marco Aiello

2024-04-17

Abstract:The recent breakthroughs in the research on Large Language Models (LLMs) have triggered a transformation across several research domains. Notably, the integration of LLMs has greatly enhanced performance in robot Task And Motion Planning (TAMP). However, previous approaches often neglect the consideration of dynamic environments, i.e., the presence of dynamic objects such as humans. In this paper, we propose a novel approach to address this gap by incorporating human awareness into LLM-based robot task planning. To obtain an effective representation of the dynamic environment, our approach integrates humans' information into a hierarchical scene graph. To ensure the plan's executability, we leverage LLMs to ground the environmental topology and actionable knowledge into formal planning language. Most importantly, we use LLMs to predict future human activities and plan tasks for the robot considering the predictions. Our contribution facilitates the development of integrating human awareness into LLM-driven robot task planning, and paves the way for proactive robot decision-making in dynamic environments.

Robotics

What problem does this paper attempt to address?

This paper attempts to address the issue of incorporating human awareness into robot task planning, especially in dynamic environments. Specifically, existing methods often overlook the presence of dynamic objects (such as humans) in dynamic environments when handling robot task and motion planning. This can lead to situations where robots may interfere with human activities or fail to effectively predict and respond to human behavior in practical applications. To solve this problem, the paper proposes a new approach that integrates human information into hierarchical scene graphs and uses large language models (LLMs) to predict future human activities, thereby achieving human-aware robot task planning. The main contributions of this approach include: 1. **Combining scene graphs and large language models**: By encoding humans and their semantic relationships with static objects into scene graphs and using LLMs to predict future human activities. 2. **Multi-agent task planning**: Treating humans as additional planning agents, using the predicted activities as their goal states, and transforming the single robot task planning problem into a multi-agent task planning problem. 3. **Improving planning performance**: By formalizing the problem into a formal planning language, ensuring that the generated task plans are executable, thereby improving planning performance. In summary, this paper aims to enable robots to make more proactive decisions and task planning in dynamic environments by incorporating human awareness, avoiding interference with human activities, and improving the efficiency and safety of task completion.

Towards Human Awareness in Robot Task Planning with Large Language Models

GG-LLM: Geometrically Grounding Large Language Models for Zero-shot Human Activity Forecasting in Human-Aware Task Planning

DELTA: Decomposed Efficient Long-Term Robot Task Planning using Large Language Models

Large Language Models for Robotics: Opportunities, Challenges, and Perspectives

Enhancing Robot Task Planning and Execution through Multi-Layer Large Language Models

Autonomous Behavior Planning For Humanoid Loco-manipulation Through Grounded Language Model

FLTRNN: Faithful Long-Horizon Task Planning for Robotics with Large Language Models

Large Language Models as Commonsense Knowledge for Large-Scale Task Planning

Action Contextualization: Adaptive Task Planning and Action Tuning using Large Language Models

One to rule them all: natural language to bind communication, perception and action

MLDT: Multi-Level Decomposition for Complex Long-Horizon Robotic Task Planning with Open-Source Large Language Model

Embodied AI in Mobile Robots: Coverage Path Planning with Large Language Models

Robot Task Planning Based on Large Language Model Representing Knowledge with Directed Graph Structures

Grounding Language Models in Autonomous Loco-manipulation Tasks

Task and Motion Planning with Large Language Models for Object Rearrangement

Nl2Hltl2Plan: Scaling Up Natural Language Understanding for Multi-Robots Through Hierarchical Temporal Logic Task Representation

LLM-SAP: Large Language Models Situational Awareness Based Planning

Long-Horizon Planning for Multi-Agent Robots in Partially Observable Environments

MHRC: Closed-loop Decentralized Multi-Heterogeneous Robot Collaboration with Large Language Models