Abstract:Robotic systems are becoming pervasive and adopted in increasingly many domains, such as manufacturing, healthcare, and space exploration. To this end, engineering software has emerged as a crucial discipline for building maintainable and reusable robotic systems. The field of robotics software engineering research has received increasing attention, fostering autonomy as a fundamental goal. However, robotics developers are still challenged trying to achieve this goal given that simulation is not able to deliver solutions to realistically emulate real-world phenomena. Robots also need to operate in unpredictable and uncontrollable environments, which require safe and trustworthy self-adaptation capabilities implemented in software. Typical techniques to address the challenges are runtime verification, field-based testing, and mitigation techniques that enable fail-safe solutions. However, there is no clear guidance to architect ROS-based systems to enable and facilitate runtime verification and field-based testing. This paper aims to fill in this gap by providing guidelines that can help developers and quality assurance (QA) teams when developing, verifying or testing their robots in the field. These guidelines are carefully tailored to address the challenges and requirements of testing robotics systems in real-world scenarios. We conducted (i) a literature review on studies addressing runtime verification and field-based testing for robotic systems, (ii) mined ROS-based applications repositories, and (iii) validated the applicability, clarity, and usefulness via two questionnaires with 55 answers overall. We contribute 20 guidelines: 8 for developers and 12 for QA teams formulated for researchers and practitioners in robotic software engineering. Finally, we map our guidelines to open challenges thus far in runtime verification and field-based testing for ROS-based systems and, we outline promising research directions in the field. Guidelines website and replication package: https://ros-rvft.github.io.

Toward the Trustworthiness of Industrial Robotics Using Differential Fuzz Testing

Enhancing ROS System Fuzzing Through Callback Tracing

Securing Autonomous Service Robots through Fuzzing, Detection, and Mitigation

RVFuzzer: Finding Input Validation Bugs in Robotic Vehicles through Control-Guided Testing

RobOT: Robustness-Oriented Testing for Deep Learning Systems

Towards Automated Acceptance testing for industrial robots

A dynamic taint tracking optimized fuzz testing method based on multi-modal sensor data fusion

Runtime Verification and Field-Based Testing for ROS-Based Robotic Systems

CORMAND2: A Deception Attack Against Industrial Robots

Vulnerability-Oriented Fuzz Testing for Connected Autonomous Vehicle Systems

IFIZZ: Deep-State and Efficient Fault-Scenario Generation to Test IoT Firmware

ROBUST: 221 Bugs in the Robot Operating System

Brief Industry Paper: Directed Kernel Fuzz Testing on Real-time Linux

Vulnerability Detection of ICS Protocols Via Cross-State Fuzzing.

An Experimental Security Analysis of an Industrial Robot Controller

Coverage-guided fuzzing for deep reinforcement learning systems

Industrial Oriented Evaluation of Fuzzing Techniques

PAVFuzz: State-Sensitive Fuzz Testing of Protocols in Autonomous Vehicles

FuzzAGG: A Fuzzing-Driven Attack Graph Generation Framework for Industrial Robot Systems