Using Normalization to Improve SMT Solver Stability

Daneshvar Amrollahi,Mathias Preiner,Aina Niemetz,Andrew Reynolds,Moses Charikar,Cesare Tinelli,Clark Barrett

2024-10-30

Abstract:In many applications, SMT solvers are used to solve similar or identical tasks over time. When the performance of the solver varies significantly despite only small changes, this leads to frustration for users. This has been called the stability problem, and it represents an important usability challenge for SMT solvers. In this paper, we introduce an approach for mitigating the stability problem based on normalizing solver inputs. We show that a perfect normalizing algorithm exists but is computationally expensive. We then describe an approximate algorithm and evaluate it on a set of benchmarks from related work, as well as a large set of benchmarks sampled from SMT-LIB. Our evaluation shows that our approximate normalizer reduces runtime variability with minimal overhead and is able to normalize a large class of mutated benchmarks to a unique normal form.

Logic in Computer Science

What problem does this paper attempt to address?

This paper attempts to solve the stability problem of SMT (Satisfiability Modulo Theories) solvers. Specifically, when there are only minor changes in the input, the performance of SMT solvers may fluctuate significantly, which brings great inconvenience to users. This phenomenon is called the "stability problem" and is an important usability challenge faced by SMT solvers in practical applications.

### Manifestations of the stability problem
1. **Performance fluctuation**: Even for queries that are semantically the same or similar, the solution times can vary greatly.
2. **Unsolvable**: Some small changes may cause problems that could originally be solved to become unsolvable.

### Causes of the problem
The stability problem mainly stems from the fact that SMT solvers use complex heuristic algorithms to solve NP - hard or more complex problems. These heuristic algorithms attempt to quickly find solutions by guiding exponential - level searches, but even very small input changes may lead to different search paths, thus affecting the solution results.

### Solutions
To alleviate this problem, the paper proposes a method based on normalization. By normalizing the solver input, semantically equivalent inputs are mapped to a standard form to reduce or eliminate performance fluctuations.

### Specific operations of normalization
The normalization process includes the following basic semantically - preserving transformations:
1. **Shuffling of assertions**
2. **Reordering of operands of commutative operators**
3. **Renaming symbols**
4. **Replacing anti - symmetric operators**

### Research questions
The paper mainly explores the following research questions:
1. Can a normalization algorithm be designed so that all benchmarks after the above transformations can be mapped to a unique output?
2. If such an algorithm exists, what is its time complexity?
3. To what extent can an efficient algorithm approximate the ideal normalization algorithm?

### Conclusions
The paper proves the existence of an ideal normalization algorithm, but its computational complexity is as difficult as the graph isomorphism problem. Therefore, the author proposes an approximation algorithm and verifies through experiments the effectiveness of this algorithm in reducing running - time fluctuations while maintaining low additional overhead.

Through this method, the paper provides an effective solution for improving the stability and usability of SMT solvers.