Abstract:Mutation testing is vital for ensuring software quality. However, the presence of equivalent mutants is known to introduce redundant cost and bias issues, hindering the effectiveness of mutation testing in practical use. Although numerous equivalent mutant detection (EMD) techniques have been proposed, they exhibit limitations due to the scarcity of training data and challenges in generalizing to unseen mutants. Recently, large language models (LLMs) have been extensively adopted in various code-related tasks and have shown superior performance by more accurately capturing program semantics. Yet the performance of LLMs in equivalent mutant detection remains largely unclear. In this paper, we conduct an empirical study on 3,302 method-level Java mutant pairs to comprehensively investigate the effectiveness and efficiency of LLMs for equivalent mutant detection. Specifically, we assess the performance of LLMs compared to existing EMD techniques, examine the various strategies of LLMs, evaluate the orthogonality between EMD techniques, and measure the time overhead of training and inference. Our findings demonstrate that LLM-based techniques significantly outperform existing techniques (i.e., the average improvement of 35.69% in terms of F1-score), with the fine-tuned code embedding strategy being the most effective. Moreover, LLM-based techniques offer an excellent balance between cost (relatively low training and inference time) and effectiveness. Based on our findings, we further discuss the impact of model size and embedding quality, and provide several promising directions for future research. This work is the first to examine LLMs in equivalent mutant detection, affirming their effectiveness and efficiency.

What problem does this paper attempt to address?

The problem that this paper attempts to solve is the effectiveness and efficiency of Equivalent Mutant Detection (EMD) in software testing. Specifically, the paper focuses on how large language models (LLMs) perform in equivalent mutant detection and whether they can outperform existing EMD techniques. ### Background and Problem Description **Background**: - **Mutation Testing** is a method for evaluating the effectiveness of a test suite by injecting artificial errors into a program. However, the existence of equivalent mutants leads to redundant costs and bias problems, affecting the practical effectiveness of mutation testing. - **Equivalent Mutants** are mutants that exhibit the same behavior as the original program in all possible test cases. These mutants cannot be identified by any test case, resulting in increased costs and result bias in mutation testing. **Problem**: - Existing equivalent mutant detection techniques have problems of data scarcity and insufficient generalization ability. - The applications of large language models (LLMs) in natural language processing and software engineering have shown superior performance, but their performance in equivalent mutant detection is still unclear. ### Paper Objectives The paper aims to evaluate the effectiveness and efficiency of LLMs in equivalent mutant detection through empirical research and answer the following research questions: 1. **RQ1**: How do the current state - of - the - art LLMs perform in equivalent mutant detection? Do LLMs have an advantage compared to existing EMD techniques? 2. **RQ2**: What is the best strategy for using LLMs for equivalent mutant detection? 3. **RQ3**: Is there orthogonality between different EMD techniques and LLM strategies? 4. **RQ4**: What is the time efficiency of the EMD techniques in the study? ### Main Contributions - **First large - scale empirical study**: The capabilities of LLMs in equivalent mutant detection are evaluated from four perspectives: effectiveness, the impact of LLM strategies, the orthogonality between different EMD techniques, and time efficiency. - **Confirmation of LLM advantages**: The study shows that LLM - based equivalent mutant detection techniques are significantly superior to existing EMD techniques and achieve state - of - the - art performance. - **Providing valuable insights**: The research results provide guidance for future research on improving LLM - based equivalent mutant detection and other software engineering fields. ### Experimental Design - **Dataset Preparation**: Use the MutantBench dataset, which contains 4,400 pairs of C/C++ and Java language mutant pairs. The research focuses on the Java language, with a total of 3,302 mutant pairs. - **Pre - processing**: Remove natural language comments and simplify mutant pair detection from the program level to the method level. - **Training and Test Dataset Construction**: Adopt a stratified sampling strategy to ensure the representativeness of the training and test datasets. ### LLMs Used Ten state - of - the - art LLMs are evaluated in the study, including: - **CodeBERT** - **GraphCodeBERT** - **PLBART** - **CodeT5** - **CodeT5+** - **UniXCoder** - **StarCoder** - **Code Llama** - **Text - Embedding Models** - **ChatGPT** ### Model Strategies - **Pre - trained Code Embedding Strategy**: Use pre - trained encoder LLMs to generate code embeddings, and then perform detection through an MLP classifier. - **Fine - tuned Code Embedding Strategy**: Update the parameters of the encoder and classifier simultaneously. - **Zero - shot Prompting Strategy**: Directly use mutant pairs and structured instructions for prompting. - **Few - shot Prompting Strategy**: Combine randomly selected examples and zero - shot prompting. - **Fine - tuning with Instructions Strategy**: Use mutant pairs with structured instructions for fine - tuning. Through these studies, the paper comprehensively evaluates the potential and practical application value of LLMs in equivalent mutant detection.

Large Language Models for Equivalent Mutant Detection: How Far Are We?

An Exploratory Study on Using Large Language Models for Mutation Testing

Equivalent Mutants Detection Based on Weighted Software Behavior Graph

LLMorpheus: Mutation Testing using Large Language Models

Mutation-based Consistency Testing for Evaluating the Code Understanding Capability of LLMs

LLMEffiChecker:Understanding and Testing Efficiency Degradation of Large Language Models

ZHANG Gongjie, XIE Chunli. Equivalent mutant analysis in weak mutation testing

LLMEffiChecker: : Understanding and Testing Efficiency Degradation of Large Language Models

A Study of Equivalent and Stubborn Mutation Operators Using Human Analysis of Equivalence

How Higher Order Mutant Testing Performs for Deep Learning Models: A Fine-Grained Evaluation of Test Effectiveness and Efficiency Improved from Second-Order Mutant-Classification Tuples

Fine-Tuning LLMs for Code Mutation: A New Era of Cyber Threats

Evaluation and Improvement of Fault Detection for Large Language Models

Effective test generation using pre-trained Large Language Models and mutation testing

VulnLLMEval: A Framework for Evaluating Large Language Models in Software Vulnerability Detection and Patching

Efficiently Predicting Protein Stability Changes Upon Single-point Mutation with Large Language Models

An Empirical Comparison of Mutant Selection Assessment Metrics

Mutagenesis screen to map the functions of parameters of Large Language Models

Understanding the Effectiveness of Large Language Models in Detecting Security Vulnerabilities

Large Language Models for Code Analysis: Do LLMs Really Do Their Job?

How Multi-Modal LLMs Reshape Visual Deep Learning Testing? A Comprehensive Study Through the Lens of Image Mutation

Deciphering the Language of Nature: A transformer-based language model for deleterious mutations in proteins