Abstract:Internet of Things (IoT) systems are becoming prevalent in various domains, from healthcare to smart homes. Testing IoT systems is critical in ensuring their reliability. Previous papers studied separately the objectives, approaches, tools, and challenges of IoT systems testing. However, despite the rapid evolution of the IoT domain, no review has been undertaken to investigate all four aspects collectively. This paper presents a systematic literature review that aggregates, synthesizes, and discusses the results of 83 primary studies (PSs) concerning IoT testing objectives, approaches, tools, and challenges. We followed the Preferred Reporting Items for Systematic Review and Meta-Analysis (PRISMA) protocol to report our findings and answer research questions (RQs). To select PSs, we applied inclusion and exclusion criteria to relevant studies published between 2012 and 2022. We extracted and analyzed the data from PSs to understand IoT systems testing. The results reveal that IoT systems testing embraces traditional software quality attributes but also introduces new ones like connectivity, energy efficiency, device lifespan, distributivity, and dynamicity. They also show that existing IoT systems testing approaches are limited to specific aspects and should be expanded for more comprehensive testing. They also show 19 testing tools and 15 testbeds for testing IoT systems with their limitations, necessitating the development or enhancement for wider coverage. The large number of heterogeneous devices generating data in different formats, along with the need for testing in real-world scenarios, poses a challenge. Thus, our study offers insights into the testing objectives, approaches, tools, and challenges associated with IoT systems. Based on the results, we also provide practical guidance for IoT practitioners by cataloging existing tools and approaches, while also identifying new research opportunities for interested researchers.

Evaluating Test Quality of Python Libraries for IoT Applications at the Network Edge

Unleashing IoT Security: Assessing the Effectiveness of Best Practices in Protecting Against Threats

Refined Characterization of Internet of Things Applications in Power Systems

A Survey on Evaluating the Quality of Autonomic Internet of Things Applications

Static analysis for discovering IoT vulnerabilities

Comparison of Code Quality and Best Practices in IoT and non-IoT Software

Automated Security Assessment for the Internet of Things

A Testbed for Experimenting Internet of Things Applications

Challenges in the Design and Implementation of IoT Testbeds in Smart-Cities: A Systematic Review

A Systematic Review of IoT Systems Testing: Objectives, Approaches, Tools, and Challenges

Quality Driven Design of Program Frameworks for Intelligent Sensor Applications

IoTInfer: Automated Blackbox Fuzz Testing of IoT Network Protocols Guided by Finite State Machine Inference

Evaluating Sensor Data Quality in Internet ofThings Smart Agriculture Applications

Data Driven Testing of Cyber Physical Systems

IoT Security and Safety Testing Toolkits for Water Distribution Systems

A comprehensive study on cybersecurity challenges and opportunities in the IoT world

Evaluating a Development Framework for Engineering Internet of Things Applications

Security Testbed for the Internet of Things

IoT Based Food Freshness Detection by Using Python

Review of Specific Features and Challenges in the Current Internet of Things Systems Impacting their Security and Reliability

The Analysis of the Behavior Patterns of Components (intelligent Sensors) in the IoT-oriented Internetware