Abstract:Robotic systems are becoming pervasive and adopted in increasingly many domains, such as manufacturing, healthcare, and space exploration. To this end, engineering software has emerged as a crucial discipline for building maintainable and reusable robotic systems. The robotics software engineering research field has received increasing attention, fostering autonomy as a fundamental goal. However, robotics developers are still challenged to achieve this goal because simulation cannot realistically deliver solutions to emulate real-world phenomena. Robots also need to operate in unpredictable and uncontrollable environments, which require safe and trustworthy self-adaptation capabilities implemented in software. Typical techniques to address the challenges are runtime verification, field-based testing, and mitigation techniques that enable fail-safe solutions. However, no clear guidance exists for architecting ROS-based systems to enable and facilitate runtime verification and field-based testing. This paper aims to fill this gap by providing guidelines to help developers and quality assurance (QA) teams develop, verify, or test their robots in the field. These guidelines are carefully tailored to address the challenges and requirements of testing robotics systems in real-world scenarios. We conducted (i) a literature review on studies addressing runtime verification and field-based testing for robotic systems, (ii) mined ROS-based applications repositories, and (iii) validated the applicability, clarity, and usefulness via two questionnaires with 55 answers overall. We contribute 20 guidelines: 8 for developers and 12 for QA teams formulated for researchers and practitioners in robotic software engineering. Finally, we map our guidelines to open challenges in runtime verification and field-based testing for ROS-based systems, and we outline promising research directions in the field.

Assuring Runtime Quality Requirements for AI-Based Components

Requirement Engineering Challenges for AI-intense Systems Development

Requirements Engineering Framework for Human-centered Artificial Intelligence Software Systems

Revising human-systems engineering principles for embedded AI applications

Quality Assurance for Artificial Intelligence: A Study of Industrial Concerns, Challenges and Best Practices

How mature is requirements engineering for AI-based systems? A systematic mapping study on practices, challenges, and future research directions

An empirical investigation of challenges of specifying training data and runtime monitors for critical software with machine learning and their relation to architectural decisions

Building Safe and Reliable AI systems for Safety Critical Tasks with Vision-Language Processing

Automated Inconsistency Analysis of Real-Time Requirements: A Domain Expert Friendly Approach.

AI‐Augmented Software Engineering: Revolutionizing or Challenging Software Quality and Testing?

Tailoring Requirements Engineering for Responsible AI

Inherent Diverse Redundant Safety Mechanisms for AI-based Software Elements in Automotive Applications

Defining Quality Requirements for a Trustworthy AI Wildflower Monitoring Platform

Correct-by-Construction Runtime Enforcement in AI -- A Survey

Reliability, Resilience and Human Factors Engineering for Trustworthy AI Systems

Runtime Verification and Field-based Testing for ROS-based Robotic Systems

Risk as a driver for AI framework development on manufacturing

Requirements Quality Assurance in Industry: Why, What and How?

Design of a Quality Management System based on the EU Artificial Intelligence Act