Abstract:The advancement of robotic systems has revolutionized numerous industries, yet their operation often demands specialized technical knowledge, limiting accessibility for non-expert users. This paper introduces ROSA (Robot Operating System Agent), an AI-powered agent that bridges the gap between the Robot Operating System (ROS) and natural language interfaces. By leveraging state-of-the-art language models and integrating open-source frameworks, ROSA enables operators to interact with robots using natural language, translating commands into actions and interfacing with ROS through well-defined tools. ROSA's design is modular and extensible, offering seamless integration with both ROS1 and ROS2, along with safety mechanisms like parameter validation and constraint enforcement to ensure secure, reliable operations. While ROSA is originally designed for ROS, it can be extended to work with other robotics middle-wares to maximize compatibility across missions. ROSA enhances human-robot interaction by democratizing access to complex robotic systems, empowering users of all expertise levels with multi-modal capabilities such as speech integration and visual perception. Ethical considerations are thoroughly addressed, guided by foundational principles like Asimov's Three Laws of Robotics, ensuring that AI integration promotes safety, transparency, privacy, and accountability. By making robotic technology more user-friendly and accessible, ROSA not only improves operational efficiency but also sets a new standard for responsible AI use in robotics and potentially future mission operations. This paper introduces ROSA's architecture and showcases initial mock-up operations in JPL's Mars Yard, a laboratory, and a simulation using three different robots. The core ROSA library is available as open-source.

User-friendly human-robot interaction based on voice commands and visual systems

Situated Multimodal Control of a Mobile Robot: Navigation through a Virtual Environment

Robot System Assistant (RoSA): Towards Intuitive Multi-Modal and Multi-Device Human-Robot Interaction

A Voice Control Platform of Mobile Robot through ROS

Human–robot interaction via voice-controllable intelligent user interface

Cross-modal Task Understanding and Execution of Voice-fingertip Reading Instruction by Using Small Family Service Robotic

Human-centered Control of a Growing Soft Robot for Object Manipulation

A multimodal interface to situated assembly robot systems

Development of a Human–Robot Interface for Cobot Trajectory Planning Using Mixed Reality

A Voice Control System for Assistive Robotic Arms: Preliminary Usability Tests on Patients

Enabling Novel Mission Operations and Interactions with ROSA: The Robot Operating System Agent

Scenario-Based Programming of Voice-Controlled Medical Robotic Systems

Human-Machine Interface for Remote Training of Robot Tasks

Pro-active service robots in a health care framework: vocal interaction using natural language and prosody

Design of a user-friendly control system for planetary rovers with CPS feature

Virtual Reality Platform to Develop and Test Applications on Human-Robot Social Interaction

Human Interface for Teleoperated Object Manipulation with a Soft Growing Robot

Human-robot-interaction using cloud-based speech recognition systems

Voice control dual arm robot based on ROS system

Development of a Human-Robot Interaction Platform for Dual-Arm Robots Based on ROS and Multimodal Artificial Intelligence

Enhancing Human–Robot Collaboration through a Multi-Module Interaction Framework with Sensor Fusion: Object Recognition, Verbal Communication, User of Interest Detection, Gesture and Gaze Recognition