A Lean Simulation Framework for Stress Testing IoT Cloud Systems

Jia Li,Behrad Moeini,Shiva Nejati,Mehrdad Sabetzadeh,Michael McCallen

2024-06-04

Abstract:The Internet of Things connects a plethora of smart devices globally across various applications like smart cities, autonomous vehicles and health monitoring. Simulation plays a key role in the testing of IoT systems, noting that field testing of a complete IoT product may be infeasible or prohibitively expensive. This paper addresses a specific yet important need in simulation-based testing for IoT: Stress testing of cloud systems. Existing stress testing solutions for IoT demand significant computational resources, making them ill-suited and costly. We propose a lean simulation framework designed for IoT cloud stress testing which enables efficient simulation of a large array of IoT and edge devices that communicate with the cloud. To facilitate simulation construction for practitioners, we develop a domain-specific language (DSL), named IoTECS, for generating simulators from model-based specifications. We provide the syntax and semantics of IoTECS and implement IoTECS using Xtext and Xtend. We assess simulators generated from IoTECS specifications for stress testing two real-world systems: a cloud-based IoT monitoring system and an IoT-connected vehicle system. Our empirical results indicate that simulators created using IoTECS: (1)achieve best performance when configured with Docker containerization; (2)effectively assess the service capacity of our case-study systems, and (3)outperform industrial stress-testing baseline tools, JMeter and Locust, by a factor of 3.5 in terms of the number of IoT and edge devices they can simulate using identical hardware resources. To gain initial insights about the usefulness of IoTECS in practice, we interviewed two engineers from our industry partner who have firsthand experience with IoTECS. Feedback from these interviews suggests that IoTECS is effective in stress testing IoT cloud systems, saving significant time and effort.

Software Engineering

What problem does this paper attempt to address?

### What problems does this paper attempt to solve?

This paper aims to solve the challenges encountered in stress - testing Internet of Things (IoT) cloud systems. Specifically, the authors propose a lightweight simulation framework for stress - testing IoT cloud systems. Traditional methods require a large amount of computing resources, making them both expensive and difficult to scale. In addition, existing stress - testing solutions do not perform well in simulating large - scale IoT devices and edge devices.

#### Main problems include:

1. **Runtime Bloat**:
- Existing IoT simulators usually capture the characteristics of each IoT device in detail, such as location, network protocol, battery capacity, etc. These detailed models result in the simulator consuming a large amount of CPU and memory resources during runtime, especially when simulating thousands of devices. Therefore, these simulators are not suitable for stress - testing.

2. **Bursty Communications**:
- When multiple IoT devices send data simultaneously, it will lead to uneven use of network bandwidth and produce communication burst phenomena. This bursty communication will affect the performance of the simulator and cannot accurately reflect the load situation in the real world.

3. **Scaling under Limited Computational Resources**:
- In practical applications, many IoT solution providers rely on hosting services for simulation. Since the simulator itself requires a large amount of computing resources, setting up and running these simulators becomes both difficult and expensive, limiting the number of IoT and edge devices that can be simulated.

To solve these problems, the authors propose a lean edge - to - cloud simulation framework with the following design features:

- **Symbolic Representation of IoT Devices**:
- Unlike existing simulators that model IoT devices as independent processes or objects, the new framework adopts a symbolic representation method, focusing only on the key attributes of IoT devices related to stress - testing, such as device load and communication methods with edge devices. This reduces resource consumption during runtime and improves the efficiency of the simulator.

- **Configurable Variables for Communication Optimization**:
- Configurable variables are introduced to adjust the startup time and data transmission interval of edge device execution, thereby minimizing communication bursts and improving the operational ability and simulation authenticity of the simulator.

- **Clustering Using Simulation Nodes**:
- The simulated edge devices are grouped into different clusters, which can be executed in native hosts, independent containers, or virtual machines. This design helps to allocate resources more efficiently, ensuring that each simulation node obtains the necessary resources according to its workload while ensuring isolation and stability.

Through these innovative designs, this framework can more effectively conduct stress - testing of large - scale IoT cloud systems, save time and cost, and provide a better user experience.