An Empirical Study of Fault Localization in Python Programs

Mohammad Rezaalipour,Carlo A. Furia

2024-03-21

Abstract:Despite its massive popularity as a programming language, especially in novel domains like data science programs, there is comparatively little research about fault localization that targets Python. Even though it is plausible that several findings about programming languages like C/C++ and Java -- the most common choices for fault localization research -- carry over to other languages, whether the dynamic nature of Python and how the language is used in practice affect the capabilities of classic fault localization approaches remain open questions to investigate. This paper is the first multi-family large-scale empirical study of fault localization on real-world Python programs and faults. Using Zou et al.'s recent large-scale empirical study of fault localization in Java as the basis of our study, we investigated the effectiveness (i.e., localization accuracy), efficiency (i.e., runtime performance), and other features (e.g., different entity granularities) of seven well-known fault-localization techniques in four families (spectrum-based, mutation-based, predicate switching, and stack-trace based) on 135 faults from 13 open-source Python projects from the BugsInPy curated collection. The results replicate for Python several results known about Java, and shed light on whether Python's peculiarities affect the capabilities of fault localization. The replication package that accompanies this paper includes detailed data about our experiments, as well as the tool FauxPy that we implemented to conduct the study.

Software Engineering

What problem does this paper attempt to address?

The problem this paper attempts to address is the effectiveness and efficiency of fault localization in Python programs. Although Python is very popular in fields such as data science, there is relatively little research on fault localization specifically for Python. Most existing fault localization research focuses on programming languages like C/C++ and Java, which have different characteristics from Python. In particular, Python's dynamic nature and its dominance in specific application domains may affect the effectiveness of classical fault localization techniques.

Specifically, this paper aims to:

1. **Evaluate the effectiveness of fault localization techniques**: Study the effectiveness of seven known fault localization techniques in four families (spectrum-based, mutation-based, predicate switching, and stack trace-based), i.e., their fault localization accuracy in real Python projects.
2. **Evaluate the efficiency of fault localization techniques**: Investigate the runtime performance of these techniques, i.e., their execution speed in real Python projects.
3. **Study the impact of different granularity levels**: Explore the performance of these techniques at different granularity levels (statement, function, and module).
4. **Analyze different types of faults and project categories**: Study the fault localization effectiveness for different types of faults and different project categories.
5. **Compare fault localization results between Python and Java**: Compare the fault localization results in Python with the findings of Zou et al. in Java to understand whether Python's characteristics affect the effectiveness of fault localization.

Through these studies, the authors hope to fill the knowledge gap in existing research and provide empirical knowledge about fault localization techniques to the Python community.