Abstract:An interactive social robotic assistant must provide services in complex and crowded spaces while adapting its behavior based on real-time human language commands or feedback. In this paper, we propose a novel hybrid approach called Social Robot Planner (SRLM), which integrates Large Language Models (LLM) and Deep Reinforcement Learning (DRL) to navigate through human-filled public spaces and provide multiple social services. SRLM infers global planning from human-in-loop commands in real-time, and encodes social information into a LLM-based large navigation model (LNM) for low-level motion execution. Moreover, a DRL-based planner is designed to maintain benchmarking performance, which is blended with LNM by a large feedback model (LFM) to address the instability of current text and LLM-driven LNM. Finally, SRLM demonstrates outstanding performance in extensive experiments. More details about this work are available at:

What problem does this paper attempt to address?

The problem that this paper attempts to solve is how, in complex and crowded human - activity spaces, social robots can, based on real - time human language instructions or feedback, provide multiple social services while adjusting their behaviors to adapt to social norms and social acceptance in dynamic environments. Specifically, the paper proposes a novel hybrid method named SRLM (Social Robot Planner), aiming to integrate large - language models (LLM) and deep reinforcement learning (DRL) to enable robots to navigate in public spaces full of humans and provide multiple social services. The main objectives of SRLM are: 1. **Understand and execute users' real - time commands**: SRLM can parse human - language input, generate global - planning guidance, and transform it into low - level action execution, thereby achieving a response to users' personalized needs. 2. **Adapt to complex dynamic optimization problems**: By combining LLM and DRL, SRLM can handle complex dynamic optimization problems in crowded environments and improve navigation performance. 3. **Enhance user experience**: SRLM not only improves navigation performance but also enhances user experience through a real - time human - machine interaction framework, allowing users to adjust the robot's behavior in real - time according to their personal feelings. 4. **Integrate high - level and low - level execution systems**: SRLM designs a high - level language navigation model (LNM) and a navigation model based on deep reinforcement learning (RLNM), and organically combines the two through a large - scale feedback model (LFM) to address the instability problems of the current text - based and LLM - driven LNM. In summary, the core problem of this paper is to develop a social - robot navigation framework that can understand and execute users' real - time commands, adapt to complex dynamic environments, enhance user experience, and integrate high - level and low - level execution systems.

SRLM: Human-in-Loop Interactive Social Robot Navigation with Large Language Model and Deep Reinforcement Learning

Enhancing Socially-Aware Robot Navigation through Bidirectional Natural Language Conversation

Speech-Guided Sequential Planning for Autonomous Navigation using Large Language Model Meta AI 3 (Llama3)

Plan-Seq-Learn: Language Model Guided RL for Solving Long Horizon Robotics Tasks

Language and Sketching: An LLM-driven Interactive Multimodal Multitask Robot Navigation Framework

LLM A: Human in the Loop Large Language Models Enabled A Search for Robotics

Language-Conditioned Offline RL for Multi-Robot Navigation

Socially Integrated Navigation: A Social Acting Robot with Deep Reinforcement Learning

VLM-Social-Nav: Socially Aware Robot Navigation through Scoring using Vision-Language Models

Deep Reinforcement Learning-based Large-scale Robot Exploration

Self-Refined Large Language Model as Automated Reward Function Designer for Deep Reinforcement Learning in Robotics

Grounding Language Models in Autonomous Loco-manipulation Tasks

Learning to Socially Navigate in Pedestrian-rich Environments with Interaction Capacity

Context-aware Mamba-based Reinforcement Learning for social robot navigation

LM-Nav: Robotic Navigation with Large Pre-Trained Models of Language, Vision, and Action

OLiVia-Nav: An Online Lifelong Vision Language Approach for Mobile Robot Social Navigation

Social Navigation with Human Empowerment driven Deep Reinforcement Learning

Role Playing Learning for Socially Concomitant Mobile Robot Navigation

Multi-Robot Cooperative Socially-Aware Navigation Using Multi-Agent Reinforcement Learning

Long-horizon Locomotion and Manipulation on a Quadrupedal Robot with Large Language Models

LGR2: Language Guided Reward Relabeling for Accelerating Hierarchical Reinforcement Learning