Abstract:This paper presents our experience specifying and verifying the correctness of ZooKeeper, a complex and evolving distributed coordination system. We use TLA+ to model fine-grained behaviors of ZooKeeper and use the TLC model checker to verify its correctness properties; we also check conformance between the model and code. The fundamental challenge is to balance the granularity of specifications and the scalability of model checking -- fine-grained specifications lead to state-space explosion, while coarse-grained specifications introduce model-code gaps. To address this challenge, we write specifications with different granularities for composable modules, and compose them into mixed-grained specifications based on specific scenarios. For example, to verify code changes, we compose fine-grained specifications of changed modules and coarse-grained specifications that abstract away details of unchanged code with preserved interactions. We show that writing multi-grained specifications is a viable practice and can cope with model-code gaps without untenable state space, especially for evolving software where changes are typically local and incremental. We detected six severe bugs that violate five types of invariants and verified their code fixes; the fixes have been merged to ZooKeeper. We also improve the protocol design to make it easy to implement correctly.

Specification and Verification with the TLA+ Trifecta: TLC, Apalache, and TLAPS

Validating Traces of Distributed Programs Against TLA+ Specifications

Specifying and Verifying PLC Systems with TLA+

Verifying Safety Properties With the TLA+ Proof System

TLC: temporal logic of distributed components

Accelerated Runtime Verification of LTL Specifications with Counting Semantics

Multi-Grained Specifications for Distributed System Model Checking and Verification

Verification Tools for Checking some kinds of Testability

Constraint-based verification of abstract models of multitreaded programs

Runtime Verification Through Forward Chaining

Constraint-based automatic verification of abstract models of multithreaded programs

Alloy meets TLA+: An exploratory study

Temporal Logic of Composable Distributed Components

Structural temporal logic for mechanized program verification

Specification sketching for Linear Temporal Logic

A Model-Checker Exploiting Structural Reductions Even With Stutter Sensitive LTL

Specifying Time-Sensitive Systems with TLA+

Sound and Complete Witnesses for Template-based Verification of LTL Properties on Polynomial Programs

Formal Verification of Digital Twins with TLA and Information Leakage Control

ProbTA: A Sound and Complete Proof Rule for Probabilistic Verification