Abstract:Internet-scale distributed systems often replicate data at multiple geographic locations to provide low latency and high availability. The Conflict-free Replicated Data Type (CRDT) is a framework that provides a principled approach to maintaining eventual consistency among data replicas. CRDTs have been notoriously difficult to design and implement correctly. Subtle deep bugs lie in the complex and tedious handling of all possible cases of conflicting data updates. We argue that the CRDT design should be formally specified and model-checked to uncover deep bugs. The implementation further needs to be systematically tested. On the one hand, the testing needs to inherit the exhaustive nature of the model checking and ensures the coverage of testing. On the other hand, the testing is expected to find coding errors which cannot be detected by design level verification. Towards the challenges above, we propose the Model Checking-driven Explorative Testing (MET) framework. At the design level, MET uses TLA+ to specify and model check CRDT designs. At the implementation level, MET conducts model checking-driven explorative testing, in the sense that the test cases are automatically generated from the model checking traces. The system execution is controlled to proceed deterministically, following the model checking trace. The explorative testing systematically controls and permutes all nondeterministic message reorderings. We apply MET in our practical development of CRDTs. The bugs in both designs and implementations of CRDTs are found. As for bugs which can be found by traditional testing techniques, MET greatly reduces the cost of fixing the bugs. Moreover, MET can find subtle deep bugs which cannot be found by existing techniques at a reasonable cost. We further discuss how MET provides us with sufficient confidence in the correctness of our CRDT designs and implementations.

Conflict-Aware Replicated Data Types

Abstraction for Conflict-Free Replicated Data Types.

Conflict-free Replicated Data Types: An Overview

Approaches to Conflict-free Replicated Data Types

Coordination-free Collaborative Replication based on Operational Transformation

VeriFx: Correct Replicated Data Types for the Masses

Remove-Win: a Design Framework for Conflict-free Replicated Data Types

Conflict-free Replicated Priority Queue: Design, Verification and Evaluation.

FabricCRDT: A Conflict-Free Replicated Datatypes Approach to Permissioned Blockchains

Collabs: A Flexible and Performant CRDT Collaboration Framework

Extending JSON CRDTs with Move Operations

Meta-operation conflict resolution for human–human interaction in collaborative feature-based CAD systems

State-Based $\infty$P-Set Conflict-Free Replicated Data Type

C5: Cloned Concurrency Control that Always Keeps Up

On Extend-Only Directed Posets and Derived Byzantine-Tolerant Replicated Data Types (Extended Version)

Consistent Query Answering for Existential Rules with Closed Predicates

Multiversion Conflict Notion for Transactional Memory Systems

Process-Commutative Distributed Objects: From Cryptocurrencies to Byzantine-Fault-Tolerant CRDTs

MET: Model Checking-Driven Explorative Testing of CRDT Designs and Implementations

Keeping CALM: When Distributed Consistency is Easy

Data Conflict Resolution Using Trust Mappings