A Comprehensive Study on Static Application Security Testing (SAST) Tools for Android

Jingyun Zhu,Kaixuan Li,Sen Chen,Lingling Fan,Junjie Wang,Xiaofei Xie
2024-10-28
Abstract:To identify security vulnerabilities in Android applications, numerous static application security testing (SAST) tools have been proposed. However, it poses significant challenges to assess their overall performance on diverse vulnerability types. The task is non-trivial and poses considerable challenges. {Firstly, the absence of a unified evaluation platform for defining and describing tools' supported vulnerability types, coupled with the lack of normalization for the intricate and varied reports generated by different tools, significantly adds to the complexity.} Secondly, there is a scarcity of adequate benchmarks, particularly those derived from real-world scenarios. To address these problems, we are the first to propose a unified platform named VulsTotal, supporting various vulnerability types, enabling comprehensive and versatile analysis across diverse SAST tools. Specifically, we begin by meticulously selecting 11 free and open-sourced SAST tools from a pool of 97 existing options, adhering to clearly defined criteria. After that, we invest significant efforts in comprehending the detection rules of each tool, subsequently unifying 67 general/common vulnerability types for {Android} SAST tools. We also redefine and implement a standardized reporting format, ensuring uniformity in presenting results across all tools. Additionally, to mitigate the problem of benchmarks, we conducted a manual analysis of huge amounts of CVEs to construct a new CVE-based benchmark based on our comprehension of Android app vulnerabilities. Leveraging the evaluation platform, which integrates both existing synthetic benchmarks and newly constructed CVE-based benchmarks from this study, we conducted a comprehensive analysis to evaluate and compare these selected tools from various perspectives, such as general vulnerability type coverage, type consistency, tool effectiveness, and time performance.
Software Engineering
What problem does this paper attempt to address?
This paper attempts to solve the following problems: 1. **Evaluating the overall performance of Android Static Application Security Testing (SAST) tools**: - Existing research lacks a comprehensive evaluation of different SAST tools on various vulnerability types, especially on fine - grained vulnerability types. - There is a lack of a unified evaluation platform to define and describe the vulnerability types supported by tools, making it difficult to make direct comparisons. - The report formats generated by different tools are complex and inconsistent, increasing the difficulty of automated comparison. 2. **Lack of sufficient benchmark tests**: - Existing benchmark tests mainly rely on synthetic data and cannot accurately reflect real - world situations. - There is a lack of benchmark tests based on real - life scenarios, especially real - vulnerability data extracted from CVE (Common Vulnerabilities and Exposures). To solve these problems, the author proposes a unified platform named **VulsTotal**, which has the following features: - **Unifying vulnerability types**: Through careful selection and analysis, the author selected 11 free and open - source SAST tools from 97 existing tools and unified 67 common vulnerability types. This enables a fair comparison of the vulnerability detection capabilities among different tools. - **Standardizing report formats**: To ensure the consistency and comparability of reports, the author re - defined and implemented a standardized report format to ensure that the results of all tools are presented in the same way. - **Constructing new benchmark tests**: To make up for the deficiencies of existing benchmark tests, the author manually analyzed a large number of CVE entries and constructed a new CVE - based benchmark test to better reflect the vulnerability situation in the real world. Through these improvements, the VulsTotal platform can more comprehensively evaluate the performance of SAST tools in different dimensions, such as vulnerability type coverage, consistency, detection effectiveness, and time performance. Finally, through a comprehensive evaluation of the 11 selected tools, the author reveals the advantages and disadvantages of current SAST tools in terms of detection ability and efficiency, and provides improvement suggestions for different stakeholders.