Abstract:Many cloud services are relying on distributed key-value stores such as ZooKeeper, Cassandra, HBase, etc. However, distributed key-value stores are notoriously difficult to design and implement without any mistakes. Because data consistency is the contract for clients that defines what the correct values to read are for a given history of operations under a specific consistency model, consistency violations can confuse client applications by showing invalid values. As a result, serious consequences such as data loss, data corruption, and unexpected behavior of client applications can occur. Software bugs are one of main reasons why consistency violations may occur. Formal verification techniques may be used to make designs correct and minimize the risks of having bugs in the implementation. However, formal verification is not a panacea due to limitations such as the cost of verification, inability to verify existing implementations, and human errors involved. Implementation-level model checking has been heavily explored by researchers for the past decades to formally verify whether the underlying implementation of distributed systems have bugs or not. Nevertheless, previous proposals are limited because their invariant checking is not versatile enough to check for the wide spectrum of consistency models, from eventual consistency to strong consistency. In this work, consistency oracles are employed for consistency invariant checking that can be used by implementation-level model checkers to formally verify data consistency model implementations of distributed key-value stores. To integrate consistency oracles with implementation-level distributed system model checkers, the partial-order information obtained via API is leveraged to avoid the exhaustive search during consistency invariant checking. Our evaluation results show that, by using the proposed method for consistency invariant checking, our prototype model checker, VConMC, can detect consistency violations caused by several real-world software bugs in a well-known distributed key-value store, ZooKeeper.

Multi-Grained Specifications for Distributed System Model Checking and Verification

Leveraging TLA+ Specifications to Improve the Reliability of the ZooKeeper Coordination Service

Automatically ‘verifying’ Discrete-Time Complex Systems Through Learning, Abstraction and Refinement

Specification and Verification of the Zab Protocol with TLA+.

Using model checker and replay facility to debug complex distributed system

Specification and Verification with the TLA+ Trifecta: TLC, Apalache, and TLAPS

Improved Reliability of Large Scale Publish/subscribe Based MOMs Using Model Checking

VConMC: Enabling Consistency Verification for Distributed Systems Using Implementation-Level Model Checkers and Consistency Oracles

Specification and Verification of Multi-clock Systems Using a Temporal Logic with Clock Constraints

Experiences from Large-Scale Model Checking: Verification of a Vehicle Control System

Validating Traces of Distributed Programs Against TLA+ Specifications

Combining Domain-Specific and Foundational Logics to Verify Complete Software Systems

An Online Model Checking Tool for Safety and Liveness Bugs

Formal Design and Verification of Zone Controller

A novel collaborative scheme of simulation and model checking for property verification

A Novel Collaborative Scheme of Simulation and Model Checking for System Properties Verification

Compositional Model Checking of Consensus Protocols Specified in TLA+ via Interaction-Preserving Abstraction

Specification and Compatibility Verification of Timing Behavior in Components

Compositional Model Checking of Consensus Protocols Via Interaction-Preserving Abstraction

Formal Specification and Verification of Complex Kernel Data Structures

Combining Model Learning and Model Checking to Analyze Java Libraries