Abstract:Robot navigation is an important research field with applications in various domains. However, traditional approaches often prioritize efficiency and obstacle avoidance, neglecting a nuanced understanding of human behavior or intent in shared spaces. With the rise of service robots, there's an increasing emphasis on endowing robots with the capability to navigate and interact in complex real-world environments. Socially aware navigation has recently become a key research area. However, existing work either predicts pedestrian movements or simply emits alert signals to pedestrians, falling short of facilitating genuine interactions between humans and robots. In this paper, we introduce the Hybrid Soft Actor-Critic with Large Language Model (HSAC-LLM), an innovative model designed for socially-aware navigation in robots. This model seamlessly integrates deep reinforcement learning with large language models, enabling it to predict both continuous and discrete actions for navigation. Notably, HSAC-LLM facilitates bidirectional interaction based on natural language with pedestrian models. When a potential collision with pedestrians is detected, the robot can initiate or respond to communications with pedestrians, obtaining and executing subsequent avoidance strategies. Experimental results in 2D simulation, the Gazebo environment, and the real-world environment demonstrate that HSAC-LLM not only efficiently enables interaction with humans but also exhibits superior performance in navigation and obstacle avoidance compared to state-of-the-art DRL algorithms. We believe this innovative paradigm opens up new avenues for effective and socially aware human-robot interactions in dynamic environments. Videos are available at <a class="link-external link-https" href="https://hsacllm.github.io/" rel="external noopener nofollow">this https URL</a>.

Enhancing human-robot communication with a comprehensive language-conditioned imitation policy for embodied robots in smart cities

Prompt, Plan, Perform: LLM-based Humanoid Control via Quantized Imitation Learning

Language-Conditioned Imitation Learning with Base Skill Priors under Unstructured Data

CLFR-M: Continual Learning Framework for Robots Via Human Feedback and Dynamic Memory

Language-Conditioned Imitation Learning for Robot Manipulation Tasks

Transferring Foundation Models for Generalizable Robotic Manipulation

Scaling Up and Distilling Down: Language-Guided Robot Skill Acquisition

Diffusion Co-Policy for Synergistic Human-Robot Collaborative Tasks

Transformer-Based Imitative Reinforcement Learning for Multirobot Path Planning

Enhancing Socially-Aware Robot Navigation through Bidirectional Natural Language Conversation

Bridging Language and Action: A Survey of Language-Conditioned Robot Manipulation

Generalized Robot Learning Framework

MHRC: Closed-loop Decentralized Multi-Heterogeneous Robot Collaboration with Large Language Models

Embodied Executable Policy Learning with Language-based Scene Summarization

EquiBot: SIM(3)-Equivariant Diffusion Policy for Generalizable and Data Efficient Learning

Humanoid robotic system for social interaction using deep imitation learning in a smart city environment

Pave the Way to Grasp Anything: Transferring Foundation Models for Universal Pick-Place Robots

Multi-modal Interaction with Transformers: Bridging Robots and Human with Natural Language

RoLD: Robot Latent Diffusion for Multi-task Policy Modeling

Data Scaling Laws in Imitation Learning for Robotic Manipulation

Autonomous Improvement of Instruction Following Skills via Foundation Models