Towards benchmarking of Solidity verification tools

Massimo Bartoletti,Fabio Fioravanti,Giulia Matricardi,Roberto Pettinau,Franco Sainas

2024-02-16

Abstract:Formal verification of smart contracts has become a hot topic in academic and industrial research, given the growing value of assets managed by decentralized applications and the consequent incentive for adversaries to tamper with them. Most of the current research on the verification of contracts revolves around Solidity, the main high-level language supported by Ethereum and other leading blockchains. Although bug detection tools for Solidity have been proliferating almost since the inception of Ethereum, only in the last few years we have seen verification tools capable of proving that a contract respects some desirable properties. An open issue is how to evaluate and compare the effectiveness of these tools: indeed, the existing benchmarks for general-purpose programming languages cannot be adapted to Solidity, given substantial differences in the programming model and in the desirable properties. We address this problem by proposing an open benchmark for Solidity verification tools. By exploiting our benchmark, we compare two leading tools, SolCMC and Certora, discussing their completeness, soundness and expressiveness limitations.

Logic in Computer Science

What problem does this paper attempt to address?

The problem that this paper attempts to solve is: how to evaluate and compare the effectiveness of Solidity smart contract verification tools. With the rapid development of decentralized applications (dApps), ensuring the security of smart contracts has become crucial. However, the existing benchmarks for general - purpose programming languages cannot be adapted to Solidity because there are significant differences in their programming models and the desired properties to be verified.

Specifically, the paper aims to:

1. **Propose a public Solidity verification tool benchmark**: The authors have developed a benchmark consisting of 323 verification tasks. Each task is composed of a Solidity smart contract and the properties it is expected to satisfy. These tasks cover multiple smart contract versions to show cases of satisfying or violating relevant properties in different ways.

2. **Evaluate the performance of existing verification tools**: By using the above - mentioned benchmark, the paper compares two leading Solidity verification tools - SolCMC and Certora, and analyzes their performance in terms of completeness, reliability, and expressiveness. In addition, a scoring scheme is introduced to quantify the performance of these tools.

### Main problems

- **Existing benchmarks are not applicable**: Traditional software verification benchmarks cannot be directly applied to Solidity because smart contracts in the blockchain environment have their unique characteristics.

- **Lack of public comparison**: Although tools such as SolCMC and Certora have been independently tested by developers, there is currently no public comparison study to evaluate their actual effectiveness.

### Solutions

- **Construct a dedicated benchmark**: The paper designs a set of benchmark test suites specifically for Solidity, including multiple versions of smart contracts and their properties, ensuring that all possible correct and incorrect scenarios are covered.

- **Automated toolchain**: An automated toolchain is provided to generate verification tasks, handle the verification process, and summarize the results, which improves the reproducibility of the experiment.

- **In - depth comparative analysis**: Through a detailed comparison of SolCMC and Certora, their respective advantages and limitations are revealed, and the reasons for inaccurate results (such as incompleteness and unreliability) are explained.

### Formula example

To ensure that the behavior of smart contracts meets expectations, formal methods are usually used for verification. For example, when verifying the deposit function, the following invariant can be defined:

\[
\text{after_deposit}(msg.sender, \text{msg.value})=\text{balances}[msg.sender]+\text{msg.value}
\]

where:
- \( \text{msg.sender} \) is the user address that initiates the transaction.
- \( \text{msg.value} \) is the amount of ETH deposited by the user.
- \( \text{balances}[msg.sender] \) is the user's balance.

In this way, it can be ensured that after each successful deposit operation, the user's balance indeed increases by the corresponding amount.

In conclusion, by constructing a comprehensive benchmark test framework, this paper provides an important reference for evaluating and improving Solidity verification tools.