Abstract:Nowadays, several software systems rely on stream processing architectures to deliver scalable performance and handle large volumes of data in near real-time. Stream processing frameworks facilitate scalable computing by distributing the application's execution across multiple machines. Despite performance being extensively studied, the measurement of fault tolerance-a key feature offered by stream processing frameworks-has still not been measured properly with updated and comprehensive testbeds. Moreover, the impact that fault recovery can have on performance is mostly ignored. This paper provides a comprehensive analysis of fault recovery performance, stability, and recovery time in a cloud-native environment with modern open-source frameworks, namely Flink, Kafka Streams, and Spark Structured Streaming. Our benchmarking analysis is inspired by chaos engineering to inject failures. Generally, our results indicate that much has changed compared to previous studies on fault recovery in distributed stream processing. In particular, the results indicate that Flink is the most stable and has one of the best fault recovery. Moreover, Kafka Streams shows performance instabilities after failures, which is due to its current rebalancing strategy that can be suboptimal in terms of load balancing. Spark Structured Streaming shows suitable fault recovery performance and stability, but with higher event latency. Our study intends to (i) help industry practitioners in choosing the most suitable stream processing framework for efficient and reliable executions of data-intensive applications; (ii) support researchers in applying and extending our research method as well as our benchmark; (iii) identify, prevent, and assist in solving potential issues in production deployments.

Can My Microservice Tolerate an Unreliable Database? Resilience Testing with Fault Injection and Visualization

MicroFI: Non-Intrusive and Prioritized Request-Level Fault Injection for Microservice Applications

When Amnesia Strikes: Understanding and Reproducing Data Loss Bugs with Fault Injection

Enhancing Fault Injection Testing of Service Systems via Fault-Tolerance Bottleneck

An Extensible Fault Tolerance Testing Framework for Microservice-Based Cloud Applications

Large-Scale Application of Fault Injection into PyTorch Models -- an Extension to PyTorchFI for Validation Efficiency

MindFI: A Fault Injection Tool for Reliability Assessment of MindSpore Applicacions

A declarative approach and benchmark tool for controlled evaluation of microservice resiliency patterns

A hybrid fault tolerance framework for SaaS services based on hidden Markov model

Using Fault Injection to Assess Blockchain Systems in Presence of Faulty Smart Contracts

Machine Learning Data Suitability and Performance Testing Using Fault Injection Testing Framework

Microusity: A testing tool for Backends for Frontends (BFF) Microservice Systems

Efficient Fuzz Testing for Apache Spark Using Framework Abstraction

Soft Error Resilience and Failure Recovery for Continuum Dynamics Applications

MicroFuzz: An Efficient Fuzzing Framework for Microservices

MicroRes: Versatile Resilience Profiling in Microservices via Degradation Dissemination Indexing

WingFuzz: Implementing Continuous Fuzzing for DBMSs.

VECROsim: A Versatile Metric-oriented Microservice Fault Simulation System (tools and Artifact Track)

A Comprehensive Benchmarking Analysis of Fault Recovery in Stream Processing Frameworks

Fault Diagnosis for Test Alarms in Microservices Through Multi-source Data

Reliable Actors with Retry Orchestration