Lessons from building static analysis tools at Google

Caitlin Sadowski,Edward Aftandilian,Alex Eagle,Liam Miller-Cushon,Ciera Jaspan

DOI: https://doi.org/10.1145/3188720

IF: 22.7

2018-03-26

Communications of the ACM

Abstract:For a static analysis project to succeed, developers must feel they benefit from and enjoy using it.

computer science, theory & methods, software engineering, hardware & architecture

What problem does this paper attempt to address?

Why Do Software Developers Use Static Analysis Tools? A User-Centered Study of Developer Needs and Motivations

Karim Ali,Lisa Nguyen Quang Do,James Wright

DOI: https://doi.org/10.1109/tse.2020.3004525

IF: 7.4

2020-01-01

IEEE Transactions on Software Engineering

Abstract:As increasingly complex software is developed every day, a growing number of companies use static analysis tools to reason about program properties ranging from simple coding style rules to more advanced software bugs, to multi-tier security vulnerabilities. While increasingly complex analyses are created, developer support must also be updated to ensure that the tools are used to their best potential. Past research in the usability of static analysis tools has primarily focused on usability issues encountered by software developers, and the causes of those issues in analysis tools. In this article, we adopt a more user-centered approach, and aim at understanding why software developers use analysis tools, which decisions they make when using those tools, what they look for when making those decisions, and the motivation behind their strategies. This approach allows us to derive new tool requirements that closely support software developers (e.g., systems for recommending warnings to fix that take developer knowledge into account), and also open novel avenues for further static-analysis research such as collaborative user interfaces for analysis warnings.

engineering, electrical & electronic,computer science, software engineering
Supporting secure programming in web applications through interactive static analysis.

Jun Zhu,Jing Xie,Heather Richter Lipford,Bill Chu

DOI: https://doi.org/10.1016/j.jare.2013.11.006

IF: 12.822

2014-01-01

Journal of Advanced Research

Abstract:Many security incidents are caused by software developers’ failure to adhere to secure programming practices. Static analysis tools have been used to detect software vulnerabilities. However, their wide usage by developers is limited by the special training required to write rules customized to application-specific logic. Our approach is interactive static analysis, to integrate static analysis into Integrated Development Environment (IDE) and provide in-situ secure programming support to help developers prevent vulnerabilities during code construction. No additional training is required nor are there any assumptions on ways programs are built. Our work is motivated in part by the observation that many vulnerabilities are introduced due to failure to practice secure programming by knowledgeable developers. We implemented a prototype interactive static analysis tool as a plug-in for Java in Eclipse. Our technical evaluation of our prototype detected multiple zero-day vulnerabilities in a large open source project. Our evaluations also suggest that false positives may be limited to a very small class of use cases.
Static Analysis Deployment Pitfalls

Flash Sheridan

DOI: https://doi.org/10.48550/arXiv.2202.13026

2022-02-26

Software Engineering

Abstract:Organizational, political, and configuration mistakes in the deployment of a static source code analysis tool within a software development organization can result in most of the value of the tool being lost, even while apparently meeting management goals. A list of pitfalls encountered as a static analysis consultant is presented, with discussion of techniques for avoiding or mitigating them. This is part of a work in progress, tentatively entitled "Handbook of Static Analysis Deployment."
Deploying Static Analysis

Flash Sheridan

DOI: https://doi.org/10.48550/arXiv.2202.11861

2022-02-24

Abstract:Static source code analysis is a powerful tool for finding and fixing bugs when deployed properly; it is, however, all too easy to deploy it in a way that looks good superficially, but which misses important defects, shows many false positives, and brings the tool into disrepute. This article is a guide to the process of deploying a static analysis tool in a large organization while avoiding the worst organizational and technical pitfalls. My main point is the importance of concentrating on the main goal of getting bugs fixed, against all the competing lesser goals which will arise during the process.

Software Engineering
Self-adaptive static analysis

Eric Bodden

DOI: https://doi.org/10.48550/arXiv.1710.07430

2017-10-20

Abstract:Static code analysis is a powerful approach to detect quality deficiencies such as performance bottlenecks, safety violations or security vulnerabilities already during a software system's implementation. Yet, as current software systems continue to grow, current static-analysis systems more frequently face the problem of insufficient scalability. We argue that this is mainly due to the fact that current static analyses are implemented fully manually, often in general-purpose programming languages such as Java or C, or in declarative languages such as Datalog. This design choice predefines the way in which the static analysis evaluates, and limits the optimizations and extensions static-analysis designers can apply. To boost scalability to a new level, we propose to fuse static-analysis with just-in-time-optimization technology, introducing for the first time static analyses that are managed and inherently self-adaptive. Those analyses automatically adapt themselves to yield a performance/precision tradeoff that is optimal with respect to the analyzed software system and to the analysis itself. Self-adaptivity is enabled by the novel idea of designing a dedicated intermediate representation, not for the analyzed program but for the analysis itself. This representation allows for an automatic optimization and adaptation of the analysis code, both ahead-of-time (through static analysis of the static analysis) as well as just-in-time during the analysis' execution, similar to just-in-time compilers.

Software Engineering
"False negative -- that one is going to kill you": Understanding Industry Perspectives of Static Analysis based Security Testing

Amit Seal Ami,Kevin Moran,Denys Poshyvanyk,Adwait Nadkarni

DOI: https://doi.org/10.1109/SP54263.2024.00019

2024-06-19

Abstract:The demand for automated security analysis techniques, such as static analysis based security testing (SAST) tools continues to increase. To develop SASTs that are effectively leveraged by developers for finding vulnerabilities, researchers and tool designers must understand how developers perceive, select, and use SASTs, what they expect from the tools, whether they know of the limitations of the tools, and how they address those limitations. This paper describes a qualitative study that explores the assumptions, expectations, beliefs, and challenges experienced by developers who use SASTs. We perform in-depth, semi-structured interviews with 20 practitioners who possess a diverse range of software development expertise, as well as a variety of unique security, product, and organizational backgrounds. We identify $17$ key findings that shed light on developer perceptions and desires related to SASTs, and also expose gaps in the status quo - challenging long-held beliefs in SAST design priorities. Finally, we provide concrete future directions for researchers and practitioners rooted in an analysis of our findings.

Cryptography and Security,Software Engineering
Static Analyzers and Potential Future Research Directions for Scala: An Overview

Eljose E Sajan,Yunpeng Zhang,Liang-Chieh Cheng

DOI: https://doi.org/10.48550/arXiv.1905.04752

2019-05-13

Abstract:Static analyzers are tool sets which are proving to be indispensable to modern programmers. These enable the programmers to detect possible errors and security defects present in the current code base within the implementation phase of the development cycle, rather than relying on a standalone testing phase. Static analyzers typically highlight possible defects within the 'static' source code and thus does not require the source code to be compiled or executed. The Scala programming language has been gaining wider adoption across various industries in recent years. With such a wide adoption of tools of this nature, this paper presents an overview on the static analysis tools available, both commercial and open-source, for the Scala programming language. This paper discusses in detail about the types of defects that each of these tools can detect, limitations of these tools and also provide potential research direction that can improve the current state of static analyzers for the Scala programming language.

Software Engineering
Efficacy of static analysis tools for software defect detection on open-source projects

Jones Yeboah,Saheed Popoola

2024-05-21

Abstract:In software practice, static analysis tools remain an integral part of detecting defects in software and there have been various tools designed to run the analysis in different programming languages like Java, C++, and Python. This paper presents an empirical comparison of popular static analysis tools for identifying software defects using several datasets using Java, C++, and Python code. The study used popular analysis tools such as SonarQube, PMD, Checkstyle, and FindBugs to perform the comparison based on using the datasets. The study also used various evaluation metrics such as Precision, Recall, and F1-score to determine the performance of each analysis tool. The study results show that SonarQube performs considerably well than all other tools in terms of its defect detection across the various three programming languages. These findings remain consistent with other existing studies that also agree on SonarQube being an effective tool for defect detection in software. The study contributes to much insight on static analysis tools with different programming languages and additional information to understand the strengths and weaknesses of each analysis tool. The study also discusses the implications for software development researchers and practitioners, and future directions in this area. Our research approach aim is to provide a recommendation guideline to enable software developers, practitioners, and researchers to make the right choice on static analysis tools to detect errors in their software codes. Also, for researchers to embark on investigating and improving software analysis tools to enhance the quality and reliability of the software systems and its software development processes practice.

Software Engineering
Customizing Static Analysis using Codesearch

Avi Hayoun,Veselin Raychev,Jack Hair

2024-04-19

Abstract:Static analysis is a growing application of software engineering, leading to a range of essential security tools, bug-finding tools, as well as software verification. Recent years show an increase of universal static analysis tools that validate a range of properties and allow customizing parts of the scanner to validate additional properties or "static analysis rules". A commonly used language to describe a range of static analysis applications is Datalog. Unfortunately, the language is still non-trivial to use, leading to analysis that is difficult to implement in a precise but performant way. In this work, we aim to make building custom static analysis tools much easier for developers, while at the same time, providing a familiar framework for application security and static analysis experts. Our approach introduces a language called StarLang, a variant of Datalog which only includes programs with a fast runtime by the means of having low time complexity of its decision procedure.

Programming Languages,Logic in Computer Science,Software Engineering
Finding and Understanding Defects in Static Analyzers by Constructing Automated Oracles

Weigang He,Peng Di,Mengli Ming,Chengyu Zhang,Ting Su,Shijie Li,Yulei Sui

DOI: https://doi.org/10.1145/3660781

2024-07-12

Abstract:Static analyzers are playing crucial roles in helping find programming mistakes and security vulnerabilities. The correctness of their analysis results is crucial for the usability in practice. Otherwise, the potential defects in these analyzers (, implementation errors, improper design choices) could affect the soundness (leading to false negatives) and precision (leading to false positives). However, finding the defects in off-the-shelf static analyzers is challenging because these analyzers usually lack clear and complete specifications, and the results of different analyzers may differ. To this end, this paper designs two novel types of automated oracles to find defects in static analyzers with randomly generated programs. The first oracle is constructed by using dynamic program executions and the second one leverages the inferred static analysis results. We applied these two oracles on three state-of-the-art static analyzers: Clang Static Analyzer (CSA), GCC Static Analyzer (GSA), and Pinpoint. We found 38 unique defects in these analyzers, 28 of which have been confirmed or fixed by the developers. We conducted a case study on these found defects followed by several insights and lessons learned for improving and better understanding static analyzers. We have made all the artifacts publicly available at https://github.com/Geoffrey1014/SA_Bugs for replication and benefit the community.
On the Benefit of Automated Static Analysis for Small and Medium-Sized Software Enterprises

Mario Gleirscher,Dmitriy Golubitskiy,Maximilian Irlbeck,Stefan Wagner

DOI: https://doi.org/10.48550/arXiv.1611.07549

2016-11-22

Software Engineering

Abstract:Today's small and medium-sized enterprises (SMEs) in the software industry are faced with major challenges. While having to work efficiently using limited resources they have to perform quality assurance on their code to avoid the risk of further effort for bug fixes or compensations. Automated static analysis can reduce this risk because it promises little effort for running an analysis. We report on our experience in analysing five projects from and with SMEs by three different static analysis techniques: code clone detection, bug pattern detection and architecture conformance analysis. We found that the effort that was needed to introduce those techniques was small (mostly below one person-hour), that we can detect diverse defects in production code and that the participating companies perceived the usefulness of the presented techniques as well as our analysis results high enough to include the techniques in their quality assurance.
A Static Analysis Platform for Investigating Security Trends in Repositories

Tim Sonnekalb,Christopher-Tobias Knaust,Bernd Gruner,Clemens-Alexander Brust,Lynn von Kurnatowski,Andreas Schreiber,Thomas S. Heinze,Patrick Mäder

2023-04-04

Abstract:Static analysis tools come in many forms andconfigurations, allowing them to handle various tasks in a (secure) development process: code style linting, bug/vulnerability detection, verification, etc., and adapt to the specific requirements of a software project, thus reducing the number of false positives.The wide range of configuration options poses a hurdle in their use for software developers, as the tools cannot be deployed out-of-the-box. However, static analysis tools only develop their full benefit if they are integrated into the software development workflow and used on regular. Vulnerability management should be integrated via version history to identify hotspots, for example. We present an analysis platform that integrates several static analysis tools that enable Git-based repositories to continuously monitor warnings across their version history. The framework is easily extensible with other tools and programming languages. We provide a visualization component in the form of a dashboard to display security trends and hotspots. Our tool can also be used to create a database of security alerts at a scale well-suited for machine learning applications such as bug or vulnerability detection.

Software Engineering
How developers engage with static analysis tools in different contexts

Carmine Vassallo,Sebastiano Panichella,Fabio Palomba,Sebastian Proksch,Harald C. Gall,Andy Zaidman

DOI: https://doi.org/10.1007/s10664-019-09750-5

IF: 3.762

2019-11-25

Empirical Software Engineering

Abstract:Automatic static analysis tools (ASATs) are instruments that support code quality assessment by automatically detecting defects and design issues. Despite their popularity, they are characterized by (i) a high false positive rate and (ii) the low comprehensibility of the generated warnings. However, no prior studies have investigated the usage of ASATs in different development contexts (e.g., code reviews, regular development), nor how open source projects integrate ASATs into their workflows. These perspectives are paramount to improve the prioritization of the identified warnings. To shed light on the actual ASATs usage practices, in this paper we first survey 56 developers (66% from industry and 34% from open source projects) and interview 11 industrial experts leveraging ASATs in their workflow with the aim of understanding how they use ASATs in different contexts. Furthermore, to investigate how ASATs are being used in the workflows of open source projects, we manually inspect the contribution guidelines of 176 open-source systems and extract the ASATs' configuration and build files from their corresponding GitHub repositories. Our study highlights that (i) 71% of developers do pay attention to different warning categories depending on the development context; (ii) 63% of our respondents rely on specific factors (e.g., team policies and composition) when prioritizing warnings to fix during their programming; and (iii) 66% of the projects define how to use specific ASATs, but only 37% enforce their usage for new contributions. The perceived relevance of ASATs varies between different projects and domains, which is a sign that ASATs use is still not a common practice. In conclusion, this study confirms previous findings on the unwillingness of developers to configure ASATs and it emphasizes the necessity to improve existing strategies for the selection and prioritization of ASATs warnings that are shown to developers.

computer science, software engineering
Improving software security with static automated code analysis in an industry setting

Dejan Baca,Bengt Carlsson,Kai Petersen,Lars Lundberg

DOI: https://doi.org/10.1002/spe.2109

2012-02-20

Abstract:SUMMARY Software security can be improved by identifying and correcting vulnerabilities. In order to reduce the cost of rework, vulnerabilities should be detected as early and efficiently as possible. Static automated code analysis is an approach for early detection. So far, only few empirical studies have been conducted in an industrial context to evaluate static automated code analysis. A case study was conducted to evaluate static code analysis in industry focusing on defect detection capability, deployment, and usage of static automated code analysis with a focus on software security. We identified that the tool was capable of detecting memory related vulnerabilities, but few vulnerabilities of other types. The deployment of the tool played an important role in its success as an early vulnerability detector, but also the developers perception of the tools merit. Classifying the warnings from the tool was harder for the developers than to correct them. The correction of false positives in some cases created new vulnerabilities in previously safe code. With regard to defect detection ability, we conclude that static code analysis is able to identify vulnerabilities in different categories. In terms of deployment, we conclude that the tool should be integrated with bug reporting systems, and developers need to share the responsibility for classifying and reporting warnings. With regard to tool usage by developers, we propose to use multiple persons (at least two) in classifying a warning. The same goes for making the decision of how to act based on the warning. Copyright © 2012 John Wiley & Sons, Ltd.
The Cost and Benefits of Static Analysis During Development

William R. Nichols Jr

DOI: https://doi.org/10.48550/arXiv.2003.03001

2020-03-06

Abstract:Without quantitative data, deciding whether and how to use static analysis in a development workflow is a matter of expert opinion and guesswork rather than an engineering trade-off. Moreover, relevant data collected under real-world conditions is scarce. Important but unknown quantitative parameters include, but are not limited to, the effort to apply the techniques, the effectiveness of removing defects, where in the workflow the analysis should be applied, and how static analysis interacts with other quality techniques. This study examined the detailed development process data 35 industrial development projects that included static analysis and that were also instrumented with the Team Software Process. We collected data project plans, logs of effort, defect, and size and post mortem reports and analyzed performance of their development activities to populate a parameterized performance model. We compared effort and defect levels with and without static analysis using a planning model that includes feedback for defect removal effectiveness and fix effort. We found evidence that using each tool developers found and removed defects at a higher rate than alternative removal techniques. Moreover, the early and inexpensive removal reduced not only final defect density but also total development effort. The contributions of this paper include real-world benchmarks of process data from projects using static analysis tools, a demonstration of a cost-effectiveness analysis using this data, and a recommendation these tools were consistently cost effective operationally.

Software Engineering
On the Use of Static Analysis to Engage Students with Software Quality Improvement: An Experience with PMD

Eman Abdullah AlOmar,Salma Abdullah AlOmar,Mohamed Wiem Mkaouer

DOI: https://doi.org/10.48550/arXiv.2302.05554

2023-02-11

Software Engineering

Abstract:Static analysis tools are frequently used to scan the source code and detect deviations from the project coding guidelines. Given their importance, linters are often introduced to classrooms to educate students on how to detect and potentially avoid these code anti-patterns. However, little is known about their effectiveness in raising students awareness, given that these linters tend to generate a large number of false positives. To increase the awareness of potential coding issues that violate coding standards, in this paper, we aim to reflect on our experience with teaching the use of static analysis for the purpose of evaluating its effectiveness in helping students with respect to improving software quality. This paper discusses the results of an experiment in the classroom over a period of 3 academic semesters, involving 65 submissions that carried out code review activity of 690 rules using PMD. The results of the quantitative and qualitative analysis shows that the presence of a set of PMD quality issues influence the acceptance or rejection of the issues, design, and best practices-related categories that take a longer time to be resolved, and students acknowledge the potential of using static analysis tools during code review. Through this experiment, code review can turn into a vital part of the educational computing plan. We envision our findings enabling educators to support students with code review strategies to raise students awareness about static analysis tools and scaffolding their coding skills.
Debugging Static Analysis

Lisa Nguyen Quang,Stefan Krüger,Patrick Hill,Karim Ali,Eric Bodden,Lisa Nguyen Quang Do,Stefan Kruger

DOI: https://doi.org/10.1109/tse.2018.2868349

IF: 7.4

2020-07-01

IEEE Transactions on Software Engineering

Abstract:Static analysis is increasingly used by companies and individual code developers to detect and fix bugs and security vulnerabilities. As programs grow more complex, the analyses have to support new code concepts, frameworks and libraries. However, static-analysis code itself is also prone to bugs. While more complex analyses are written and used in production systems every day, the cost of debugging and fixing them also increases tremendously. To understand the difficulties of debugging static analysis, we surveyed 115 static-analysis writers. From their responses, we determined the core requirements to build a debugger for static analyses, which revolve around two main issues: abstracting from both the analysis code and the code it analyses at the same time, and tracking the analysis internal state throughout both code bases. Most tools used by our survey participants lack the capabilities to address both issues. Focusing on those requirements, we introduce Visuflow, a debugging environment for static data-flow analysis. Visuflow features graph visualizations and custom breakpoints that enable users to view the state of an analysis at any time. In a user study on 20 static-analysis writers, Visuflow helped identify 25 and fix 50 percent more errors in the analysis code compared to the standard Eclipse debugging environment.
Interactive Static Software Performance Analysis in the IDE

Aaron Beigelbeck,Maurício Aniche,Jürgen Cito

DOI: https://doi.org/10.48550/arXiv.2105.02023

2021-05-04

Software Engineering

Abstract:Detecting performance issues due to suboptimal code during the development process can be a daunting task, especially when it comes to localizing them after noticing performance degradation after deployment. Static analysis has the potential to provide early feedback on performance problems to developers without having to run profilers with expensive (and often unavailable) performance tests. We develop a VSCode tool that integrates the static performance analysis results from Infer via code annotations and decorations (surfacing complexity analysis results in context) and side panel views showing details and overviews (enabling explainability of the results). Additionally, we design our system for interactivity to allow for more responsiveness to code changes as they happen. We evaluate the efficacy of our tool by measuring the overhead that the static performance analysis integration introduces in the development workflow. Further, we report on a case study that illustrates how our system can be used to reason about software performance in the context of a real performance bug in the ElasticSearch open-source project. Demo video: https://www.youtube.com/watch?v=-GqPb_YZMOs Repository: https://github.com/ipa-lab/vscode-infer-performance
How far are German companies in improving security through static program analysis tools?

Goran Piskachev,Stefan Dziwok,Thorsten Koch,Sven Merschjohan,Eric Bodden

DOI: https://doi.org/10.48550/arXiv.2208.06136

2022-08-12

Abstract:As security becomes more relevant for many companies, the popularity of static program analysis (SPA) tools is increasing. In this paper, we target the use of SPA tools among companies in Germany with a focus on security. We give insights on the current issues and the developers' willingness to configure the tools to overcome these issues. Compared to previous studies, our study considers the companies' culture and processes for using SPA tools. We conducted an online survey with 256 responses and semi-structured interviews with 17 product owners and executives from multiple companies. Our results show a diversity in the usage of tools. Only half of our survey participants use SPA tools. The free tools tend to be more popular among software developers. In most companies, software developers are encouraged to use free tools, whereas commercial tools can be requested. However, the product owners and executives in our interviews reported that their developers do not request new tools. We also find out that automatic security checks with tools are rarely performed on each release.

Cryptography and Security,Software Engineering
Tai-e: A Developer-Friendly Static Analysis Framework for Java by Harnessing the Good Designs of Classics.

Tian Tan,Yue Li

DOI: https://doi.org/10.1145/3597926.3598120

2022-01-01

Abstract:Static analysis is a mature field with applications to bug detection, security analysis, program understanding, optimization, and more. To facilitate these applications, static analysis frameworks play an essential role by providing a series of fundamental services such as intermediate representation (IR) generation, control flow graph construction, points-to/alias information computation, and so on. However, although static analysis has made great strides and several well-known frameworks have emerged in this field over the past decades, these frameworks are not that easy to learn and use for developers who rely on them to create and implement analyses. In that sense, it is far from trivial to build a developer-friendly static analysis framework, because compared to the knowledge required for static analysis itself, we have significantly less knowledge designing and implementing static analysis frameworks. In this work, we take a step forward by discussing the design trade-offs for the crucial components of a static analysis framework for Java, and select the designs by following the HGDC (Harnessing the Good Designs of Classics) principle: for each crucial component of a static analysis framework, we compare the design choices made for it (possibly) by different classic frameworks such as Soot, Wala, Doop, SpotBugs and Checker, and choose arguably a more appropriate one; but if none is good enough, we then propose a better design. These selected or newly proposed designs finally constitute Tai-e, a new static analysis framework for Java, which has been implemented from scratch. Tai-e is novel in the designs of several aspects like IR, pointer analysis and development of new analyses, etc., leading to a developer-friendly (easy-to-learn and easy-to-use) analysis framework. To the best of our knowledge, this is the first work that systematically explores the designs and implementations of various static analysis frameworks for Java. We expect it to provide useful materials and viewpoints for building better static analysis infrastructures, and we hope that it could draw more attentions of the community to this challenging but tangible topic.

Lessons from building static analysis tools at Google

Why Do Software Developers Use Static Analysis Tools? A User-Centered Study of Developer Needs and Motivations

Supporting secure programming in web applications through interactive static analysis.

Static Analysis Deployment Pitfalls

Deploying Static Analysis

Self-adaptive static analysis

"False negative -- that one is going to kill you": Understanding Industry Perspectives of Static Analysis based Security Testing

Static Analyzers and Potential Future Research Directions for Scala: An Overview

Efficacy of static analysis tools for software defect detection on open-source projects

Customizing Static Analysis using Codesearch

Finding and Understanding Defects in Static Analyzers by Constructing Automated Oracles

On the Benefit of Automated Static Analysis for Small and Medium-Sized Software Enterprises

A Static Analysis Platform for Investigating Security Trends in Repositories

How developers engage with static analysis tools in different contexts

Improving software security with static automated code analysis in an industry setting

The Cost and Benefits of Static Analysis During Development

On the Use of Static Analysis to Engage Students with Software Quality Improvement: An Experience with PMD

Debugging Static Analysis

Interactive Static Software Performance Analysis in the IDE

How far are German companies in improving security through static program analysis tools?

Tai-e: A Developer-Friendly Static Analysis Framework for Java by Harnessing the Good Designs of Classics.