On the Relationship between Code Verifiability and Understandability

Kobi Feldman,Martin Kellogg,Oscar Chaparro

DOI: https://doi.org/10.1145/3611643.3616242

2023-10-31

Abstract:Proponents of software verification have argued that simpler code is easier to verify: that is, that verification tools issue fewer false positives and require less human intervention when analyzing simpler code. We empirically validate this assumption by comparing the number of warnings produced by four state-of-the-art verification tools on 211 snippets of Java code with 20 metrics of code comprehensibility from human subjects in six prior studies. Our experiments, based on a statistical (meta-)analysis, show that, in aggregate, there is a small correlation (r = 0.23) between understandability and verifiability. The results support the claim that easy-to-verify code is often easier to understand than code that requires more effort to verify. Our work has implications for the users and designers of verification tools and for future attempts to automatically measure code comprehensibility: verification tools may have ancillary benefits to understandability, and measuring understandability may require reasoning about semantic, not just syntactic, code properties.

Software Engineering

What problem does this paper attempt to address?

The paper aims to explore the relationship between code verifiability and understandability. Specifically, researchers have validated a common assumption through empirical analysis: simpler code is not only easier for verification tools to verify (i.e., verification tools produce fewer false positives and require less human intervention when analyzing simple code), but it is also easier for developers to understand. To test this hypothesis, they compared the number of warnings generated by four state-of-the-art code verification tools on 211 Java code snippets, which encompassed 20 types of code understandability metrics assessed by human subjects from six prior studies.

The research results indicate that, overall, there is a weak correlation between code understandability and verifiability (r=0.23). This supports the view that code that is easy to verify is often easier to understand than code that requires more effort to verify. This finding has important implications for users and designers of verification tools, as well as for future attempts to automatically measure code understandability. It suggests that verification tools may have additional benefits for code understandability, and that measuring code understandability may need to consider semantic properties, not just syntactic properties.

The main contribution of this paper is that it provides empirical evidence of the correlation between code understandability and verifiability, supporting the common assumption that code that is easy to verify is also easier for humans to understand. Additionally, it provides an online replication package, enabling the validation of research results and future scientific work.