Language-Based Software Testing

Dominic Steinhöfel,Andreas Zeller

DOI: https://doi.org/10.1145/3631520

IF: 22.7

2024-03-25

Communications of the ACM

Abstract:Constraints over grammar elements can make test generation easier than ever.

computer science, theory & methods, software engineering, hardware & architecture

What problem does this paper attempt to address?

Random Grammar-based Testing for Covering All Non-Terminals

Alois Dreyfus,Pierre-Cyrille Heam,Olga Kouchnarenko

DOI: https://doi.org/10.48550/arXiv.1311.6606

2013-11-26

Abstract:In the context of software testing, generating complex data inputs is frequently performed using a grammar-based specification. For combinatorial reasons, an exhaustive generation of the data -- of a given size -- is practically impossible, and most approaches are either based on random techniques or on coverage criteria. In this paper, we show how to combine these two techniques by biasing the random generation in order to optimise the probability of satisfying a coverage criterion.

Software Engineering
Large Language Models to Generate System-Level Test Programs Targeting Non-functional Properties

Denis Schwachhofer,Peter Domanski,Steffen Becker,Stefan Wagner,Matthias Sauer,Dirk Pflüger,Ilia Polian

2024-03-19

Abstract:System-Level Test (SLT) has been a part of the test flow for integrated circuits for over a decade and still gains importance. However, no systematic approaches exist for test program generation, especially targeting non-functional properties of the Device under Test (DUT). Currently, test engineers manually compose test suites from off-the-shelf software, approximating the end-user environment of the DUT. This is a challenging and tedious task that does not guarantee sufficient control over non-functional properties. This paper proposes Large Language Models (LLMs) to generate test programs. We take a first glance at how pre-trained LLMs perform in test program generation to optimize non-functional properties of the DUT. Therefore, we write a prompt to generate C code snippets that maximize the instructions per cycle of a super-scalar, out-of-order architecture in simulation. Additionally, we apply prompt and hyperparameter optimization to achieve the best possible results without further training.

Software Engineering,Artificial Intelligence,Emerging Technologies,Programming Languages
Automatic Grammar-Based Test Generation.

Hai-Feng Guo,Zongyan Qiu

DOI: https://doi.org/10.1007/978-3-642-41707-8_2

2013-01-01

Abstract:In this paper, we present an automatic grammar-based test generation approach which takes a symbolic grammar as input, requires zero control input from users, and produces well-distributed test cases. Our approach utilizes a novel dynamic stochastic model where each variable is associated with a tuple of probability distributions, which are dynamically adjusted along the derivation. The adjustment is based on a tabling strategy to keep track of the recursion of each grammar variable. We further present a test generation coverage tree illustrating the distribution of generated test cases and their detailed derivations, more importantly, it provides various implicit balance control mechanisms. We implemented this approach in a Java-based system, named Gena. Experimental results demonstrate the effectiveness of our test generation approach and show the balanced distribution of generated test cases over grammatical structures. © IFIP International Federation for Information Processing 2013.
A Dynamic Stochastic Model for Automatic Grammar-Based Test Generation

Hai-Feng Guo,Zongyan Qiu

DOI: https://doi.org/10.1002/spe.2278

2014-01-01

Abstract:SummaryGrammar‐based test generation provides a systematic approach to producing test cases from a given context‐free grammar. Unfortunately, naive grammar‐based test generation is problematic because of the fact that exhaustive random test case production is often explosive, and grammar‐based test generation with explicit annotation controls often causes unbalanced testing coverage. In this paper, we present an automatic grammar‐based test generation approach, which takes a symbolic grammar as input, requires zero control input from users, and produces well‐distributed test cases. Our approach utilizes a novel dynamic stochastic model where each variable is associated with a tuple of probability distributions, which are dynamically adjusted along the derivation. We further present a coverage tree illustrating the distribution of generated test cases and their detailed derivations. More importantly, the coverage tree supports various implicit derivation control mechanisms. We implemented this approach in a Java‐based system, named Gena. Each test case generated by Gena automatically comes with a set of structural features, which can play an important and effective role on automated failure causes localization. Experimental results demonstrate the effectiveness of our approach, the well‐balanced distribution of generated test cases over grammatical structures, and a case study on grammar‐based failure causes localization. Copyright © 2014 John Wiley & Sons, Ltd.
Testing by Dualization

Yishuai Li

DOI: https://doi.org/500.14332/32046

2022-10-04

Abstract:Software engineering requires rigorous testing to guarantee the product's quality. Semantic testing of functional correctness is challenged by nondeterminism in behavior, which makes testers difficult to write and reason about. This thesis presents a language-based technique for testing interactive systems. I propose a theory for specifying and validating nondeterministic behaviors, with guaranteed soundness and correctness. I then apply the theory to testing practices, and show how to derive specifications into interactive tester programs. I also introduce a language design for producing test inputs that can effectively detect and reproduce invalid behaviors. I evaluate the methodology by specifying and testing real-world systems such as web servers and file synchronizers, demonstrating the derived testers' ability to find disagreements between the specification and the implementation.

Programming Languages,Software Engineering
The Use of Instrumentation in Grammar Engineering

Norbert Broeker

DOI: https://doi.org/10.48550/arXiv.cs/0011020

2000-11-30

Abstract:This paper explores the usefulness of a technique from software engineering, code instrumentation, for the development of large-scale natural language grammars. Information about the usage of grammar rules in test and corpus sentences is used to improve grammar and testsuite, as well as adapting a grammar to a specific genre. Results show that less than half of a large-coverage grammar for German is actually tested by two large testsuites, and that 10--30% of testing time is redundant. This methodology applied can be seen as a re-use of grammar writing knowledge for testsuite compilation.

Computation and Language
Software Testing with Large Language Models: Survey, Landscape, and Vision

Junjie Wang,Yuchao Huang,Chunyang Chen,Zhe Liu,Song Wang,Qing Wang

DOI: https://doi.org/10.48550/arXiv.2307.07221

2024-03-04

Abstract:Pre-trained large language models (LLMs) have recently emerged as a breakthrough technology in natural language processing and artificial intelligence, with the ability to handle large-scale datasets and exhibit remarkable performance across a wide range of tasks. Meanwhile, software testing is a crucial undertaking that serves as a cornerstone for ensuring the quality and reliability of software products. As the scope and complexity of software systems continue to grow, the need for more effective software testing techniques becomes increasingly urgent, making it an area ripe for innovative approaches such as the use of LLMs. This paper provides a comprehensive review of the utilization of LLMs in software testing. It analyzes 102 relevant studies that have used LLMs for software testing, from both the software testing and LLMs perspectives. The paper presents a detailed discussion of the software testing tasks for which LLMs are commonly used, among which test case preparation and program repair are the most representative. It also analyzes the commonly used LLMs, the types of prompt engineering that are employed, as well as the accompanied techniques with these LLMs. It also summarizes the key challenges and potential opportunities in this direction. This work can serve as a roadmap for future research in this area, highlighting potential avenues for exploration, and identifying gaps in our current understanding of the use of LLMs in software testing.

Software Engineering
LLM for Test Script Generation and Migration: Challenges, Capabilities, and Opportunities

Shengcheng Yu,Chunrong Fang,Yuchen Ling,Chentian Wu,Zhenyu Chen

2023-09-24

Abstract:This paper investigates the application of large language models (LLM) in the domain of mobile application test script generation. Test script generation is a vital component of software testing, enabling efficient and reliable automation of repetitive test tasks. However, existing generation approaches often encounter limitations, such as difficulties in accurately capturing and reproducing test scripts across diverse devices, platforms, and applications. These challenges arise due to differences in screen sizes, input modalities, platform behaviors, API inconsistencies, and application architectures. Overcoming these limitations is crucial for achieving robust and comprehensive test automation. By leveraging the capabilities of LLMs, we aim to address these challenges and explore its potential as a versatile tool for test automation. We investigate how well LLMs can adapt to diverse devices and systems while accurately capturing and generating test scripts. Additionally, we evaluate its cross-platform generation capabilities by assessing its ability to handle operating system variations and platform-specific behaviors. Furthermore, we explore the application of LLMs in cross-app migration, where it generates test scripts across different applications and software environments based on existing scripts. Throughout the investigation, we analyze its adaptability to various user interfaces, app architectures, and interaction patterns, ensuring accurate script generation and compatibility. The findings of this research contribute to the understanding of LLMs' capabilities in test automation. Ultimately, this research aims to enhance software testing practices, empowering app developers to achieve higher levels of software quality and development efficiency.

Software Engineering
SpecTest: Specification-Based Compiler Testing

Richard Schumi,Jun Sun

DOI: https://doi.org/10.1007/978-3-030-71500-7_14

2021-01-01

Abstract:Abstract Compilers are error-prone due to their high complexity. They are relevant for not only general purpose programming languages, but also for many domain specific languages. Bugs in compilers can potentially render all programs at risk. It is thus crucial that compilers are systematically tested, if not verified. Recently, a number of efforts have been made to formalise and standardise programming language semantics, which can be applied to verify the correctness of the respective compilers. In this work, we present a novel specification-based testing method named SpecTest to better utilise these semantics for testing. By applying an executable semantics as test oracle, SpecTest can discover deep semantic errors in compilers. Compared to existing approaches, SpecTest is built upon a novel test coverage criterion called semantic coverage which brings together mutation testing and fuzzing to specifically target less tested language features. We apply SpecTest to systematically test two compilers, i.e., the Java compiler and the Solidity compiler. SpecTest improves the semantic coverage of both compilers considerably and reveals multiple previously unknown bugs.
Leveraging Large Language Models for Enhancing the Understandability of Generated Unit Tests

Amirhossein Deljouyi,Roham Koohestani,Maliheh Izadi,Andy Zaidman

2024-08-21

Abstract:Automated unit test generators, particularly search-based software testing tools like EvoSuite, are capable of generating tests with high coverage. Although these generators alleviate the burden of writing unit tests, they often pose challenges for software engineers in terms of understanding the generated tests. To address this, we introduce UTGen, which combines search-based software testing and large language models to enhance the understandability of automatically generated test cases. We achieve this enhancement through contextualizing test data, improving identifier naming, and adding descriptive comments. Through a controlled experiment with 32 participants from both academia and industry, we investigate how the understandability of unit tests affects a software engineer's ability to perform bug-fixing tasks. We selected bug-fixing to simulate a real-world scenario that emphasizes the importance of understandable test cases. We observe that participants working on assignments with UTGen test cases fix up to 33% more bugs and use up to 20% less time when compared to baseline test cases. From the post-test questionnaire, we gathered that participants found that enhanced test names, test data, and variable names improved their bug-fixing process.

Software Engineering,Artificial Intelligence
Assessing Language Testing: A Corpus-Based Approach

Youxia Sun,Lianzhen He

2015-01-01

Abstract:Language Testing has now completed three decades of publication, during which a great amount of significant work has been done. In this paper, a corpus of 30 volumes of this journal (1984-2013) is built to analyze the research foci and changes, research methodologies, representative scholars and research centers that have long enjoyed cachet. Based on the quantitative and qualitative analysis of these data, it can be concluded that this journal is representative of the development in the field of language testing, and it has been and is reflecting and promoting research, as well as changes in this field and beyond over the years.
Grammar-based test suite construction using coverage-directed algorithms over LR-graphs

Christoff Rossouw,Bernd Fischer

DOI: https://doi.org/10.2139/ssrn.4423845

IF: 3.5

2024-04-26

Journal of Systems and Software

Abstract:In grammar-based testing, the test suites that drive the system under test are typically constructed from a given context-free grammar through a set of derivations that jointly satisfy some coverage criterion. In this paper, we describe and evaluate a new algorithm that instead constructs test suites from a set of valid paths that cover all edges in a labeled directed graph corresponding to an LR-automaton that accepts the language of the grammar. Vertices in this graph correspond to states in the LR-automaton; two vertices are connected by an edge iff the top of the LR-automaton's stack can change from one state to the other, either by shifting a terminal or non-terminal symbol (push edges), or by reducing with a grammar rule (pop edges). The algorithm constructs a unique reduction path for each pop edge in the graph. These reduction paths are recursively embedded into each other, and any unresolved non-terminal push edges are substituted by shortest derivations for the non-terminal symbol. The algorithm can work with different types of LR-automata, including LR(0)- and LR(1)-automata, and can successfully generate a test suite from an LR-graph even if the underlying LR-automaton construction leads to shift/reduce or reduce/reduce conflicts. The algorithm only constructs valid paths over the LR-graphs that correspond to sentences in the language and thus generates only positive tests. We therefore also describe mutations to the positive paths that are guaranteed to generate negative tests without needing any further verification by an oracle. Our algorithm is substantially more efficient than an earlier algorithm that explores LR-graphs with two consecutive breadth-first graph traversals and our experimental evaluation shows that it scales to large production-quality grammars. It is robust against random choices made to resolve ambiguity in the construction of the tests, while the code coverage of the different test suite variants is relatively uniform. Finally, our evaluation shows that the negative test suites constructed by path mutation identify more faults in a set of student grammars than those constructed by rule mutation.

computer science, theory & methods, software engineering
Automated Test Oracle Generation via Denotational Semantics

Hai-Feng Guo,Liang Cao,Yushu Song,Zongyan Qiu

DOI: https://doi.org/10.1109/QSIC.2014.38

2014-01-01

Abstract:This paper presents a formal approach to specifying test oracles in denotational semantics, and bridges the automation between test case generation and software testing. We extended our grammar-based test generation tool, Gena, with an interface for users to define semantic domains, and formal semantic evaluation functions for mapping from syntax to semantics, based on the methodology of denotational semantics. We provide two practical case studies to illustrate how automated oracle generation can be effectively integrated with grammar-based test generation in different testing scenarios: generating test data for a set of java programs, and generating executable Smalltalk testing methods.
A System for Automated Unit Test Generation Using Large Language Models and Assessment of Generated Test Suites

Andrea Lops,Fedelucio Narducci,Azzurra Ragone,Michelantonio Trizio,Claudio Bartolini

2024-08-16

Abstract:Unit tests represent the most basic level of testing within the software testing lifecycle and are crucial to ensuring software correctness. Designing and creating unit tests is a costly and labor-intensive process that is ripe for automation. Recently, Large Language Models (LLMs) have been applied to various aspects of software development, including unit test generation. Although several empirical studies evaluating LLMs' capabilities in test code generation exist, they primarily focus on simple scenarios, such as the straightforward generation of unit tests for individual methods. These evaluations often involve independent and small-scale test units, providing a limited view of LLMs' performance in real-world software development scenarios. Moreover, previous studies do not approach the problem at a suitable scale for real-life applications. Generated unit tests are often evaluated via manual integration into the original projects, a process that limits the number of tests executed and reduces overall efficiency. To address these gaps, we have developed an approach for generating and evaluating more real-life complexity test suites. Our approach focuses on class-level test code generation and automates the entire process from test generation to test assessment. In this work, we present AgoneTest: an automated system for generating test suites for Java projects and a comprehensive and principled methodology for evaluating the generated test suites. Starting from a state-of-the-art dataset (i.e., Methods2Test), we built a new dataset for comparing human-written tests with those generated by LLMs. Our key contributions include a scalable automated software system, a new dataset, and a detailed methodology for evaluating test quality.

Software Engineering,Artificial Intelligence
SWT-Bench: Testing and Validating Real-World Bug-Fixes with Code Agents

Niels Mündler,Mark Niklas Müller,Jingxuan He,Martin Vechev

2024-11-17

Abstract:Rigorous software testing is crucial for developing and maintaining high-quality code, making automated test generation a promising avenue for both improving software quality and boosting the effectiveness of code generation methods. However, while code generation with Large Language Models (LLMs) is an extraordinarily active research area, test generation remains relatively unexplored. We address this gap and investigate the capability of LLM-based Code Agents to formalize user issues into test cases. To this end, we propose a novel benchmark based on popular GitHub repositories, containing real-world issues, ground-truth bug-fixes, and golden tests. We find that LLMs generally perform surprisingly well at generating relevant test cases, with Code Agents designed for code repair exceeding the performance of systems designed specifically for test generation. Further, as test generation is a similar but more structured task than code generation, it allows for a more fine-grained analysis using issue reproduction rate and coverage changes, providing a dual metric for analyzing systems designed for code repair. Finally, we find that generated tests are an effective filter for proposed code fixes, doubling the precision of SWE-Agent. We release all data and code at <a class="link-external link-https" href="https://github.com/logic-star-ai/SWT-Bench" rel="external noopener nofollow">this https URL</a>

Software Engineering,Artificial Intelligence,Machine Learning
Search-based Testing for Scratch Programs

Adina Deiner,Christoph Frädrich,Gordon Fraser,Sophia Geserer,Niklas Zantner

DOI: https://doi.org/10.48550/arXiv.2009.04115

2020-09-09

Software Engineering

Abstract:Block-based programming languages enable young learners to quickly implement fun programs and games. The Scratch programming environment is particularly successful at this, with more than 50 million registered users at the time of this writing. Although Scratch simplifies creating syntactically correct programs, learners and educators nevertheless frequently require feedback and support. Dynamic program analysis could enable automation of this support, but the test suites necessary for dynamic analysis do not usually exist for Scratch programs. It is, however, possible to cast test generation for Scratch as a search problem. In this paper, we introduce an approach for automatically generating test suites for Scratch programs using grammatical evolution. The use of grammatical evolution clearly separates the search encoding from framework-specific implementation details, and allows us to use advanced test acceleration techniques. We implemented our approach as an extension of the Whisker test framework. Evaluation on sample Scratch programs demonstrates the potential of the approach.
Are We Testing or Being Tested? Exploring the Practical Applications of Large Language Models in Software Testing

Robson Santos,Italo Santos,Cleyton Magalhaes,Ronnie de Souza Santos

DOI: https://doi.org/10.48550/arXiv.2312.04860

2023-12-08

Software Engineering

Abstract:A Large Language Model (LLM) represents a cutting-edge artificial intelligence model that generates coherent content, including grammatically precise sentences, human-like paragraphs, and syntactically accurate code snippets. LLMs can play a pivotal role in software development, including software testing. LLMs go beyond traditional roles such as requirement analysis and documentation and can support test case generation, making them valuable tools that significantly enhance testing practices within the field. Hence, we explore the practical application of LLMs in software testing within an industrial setting, focusing on their current use by professional testers. In this context, rather than relying on existing data, we conducted a cross-sectional survey and collected data within real working contexts, specifically, engaging with practitioners in industrial settings. We applied quantitative and qualitative techniques to analyze and synthesize our collected data. Our findings demonstrate that LLMs effectively enhance testing documents and significantly assist testing professionals in programming tasks like debugging and test case automation. LLMs can support individuals engaged in manual testing who need to code. However, it is crucial to emphasize that, at this early stage, software testing professionals should use LLMs with caution while well-defined methods and guidelines are being built for the secure adoption of these tools.
Language testing

DOI: https://doi.org/10.1017/s0261444802241816

2002-07-01

Language Teaching

Abstract:02–517 Brindley, Geoff (Macquarie U., New South Wales, Australia; Email: gbrindle@laurel.ocs.mq.edu.au). Outcomes-based assessment in practice: Some examples and emerging insights. Language Testing (London, UK), 18, 4 (2001), 393–407.

education & educational research,linguistics
Adaptive Test Generation Using a Large Language Model

Max Schäfer,Sarah Nadi,Aryaz Eghbali,Frank Tip

DOI: https://doi.org/10.48550/arXiv.2302.06527

2023-02-13

Software Engineering

Abstract:Unit tests play a key role in ensuring the correctness of software. However, manually creating unit tests is a laborious task, motivating the need for automation. This paper presents TestPilot, an adaptive test generation technique that leverages Large Language Models (LLMs). TestPilot uses Codex, an off-the-shelf LLM, to automatically generate unit tests for a given program without requiring additional training or few-shot learning on examples of existing tests. In our approach, Codex is provided with prompts that include the signature and implementation of a function under test, along with usage examples extracted from documentation. If a generated test fails, TestPilot's adaptive component attempts to generate a new test that fixes the problem by re-prompting the model with the failing test and error message. We created an implementation of TestPilot for JavaScript and evaluated it on 25 npm packages with a total of 1,684 API functions to generate tests for. Our results show that the generated tests achieve up to 93.1% statement coverage (median 68.2%). Moreover, on average, 58.5% of the generated tests contain at least one assertion that exercises functionality from the package under test. Our experiments with excluding parts of the information included in the prompts show that all components contribute towards the generation of effective test suites. Finally, we find that TestPilot does not generate memorized tests: 92.7% of our generated tests have $\leq$ 50% similarity with existing tests (as measured by normalized edit distance), with none of them being exact copies.
Software testing with code-based test generators: data and lessons learned from a case study with an industrial software component

Pietro Braione,Giovanni Denaro,Andrea Mattavelli,Mattia Vivanti,Ali Muhammad

DOI: https://doi.org/10.1007/s11219-013-9207-1

2013-08-02

Software Quality Journal

Abstract:Automatically generating effective test suites promises a significant impact on testing practice by promoting extensively tested software within reasonable effort and cost bounds. Code-based test generators rely on the source code of the software under test to identify test objectives and to steer the test case generation process accordingly. Currently, the most mature proposals on this topic come from the research on random testing, dynamic symbolic execution, and search-based testing. This paper studies the effectiveness of a set of state-of-the-research test generators on a family of industrial programs with nontrivial domain-specific peculiarities. These programs are part of a software component of a real-time and safety-critical control system and integrate in a control task specified in LabVIEW, a graphical language for designing embedded systems. The result of this study enhances the available body of knowledge on the strengths and weaknesses of test generators. The empirical data indicate that the test generators can truly expose subtle (previously unknown) bugs in the subject software and that there can be merit in using different types of test generation approaches in a complementary, even synergic fashion. Furthermore, our experiment pinpoints the support for floating point arithmetics and nonlinear computations as a major milestone in the path to exploiting the full potential of the prototypes based on symbolic execution in industry.

computer science, software engineering

Language-Based Software Testing

Random Grammar-based Testing for Covering All Non-Terminals

Large Language Models to Generate System-Level Test Programs Targeting Non-functional Properties

Automatic Grammar-Based Test Generation.

A Dynamic Stochastic Model for Automatic Grammar-Based Test Generation

Testing by Dualization

The Use of Instrumentation in Grammar Engineering

Software Testing with Large Language Models: Survey, Landscape, and Vision

LLM for Test Script Generation and Migration: Challenges, Capabilities, and Opportunities

SpecTest: Specification-Based Compiler Testing

Leveraging Large Language Models for Enhancing the Understandability of Generated Unit Tests

Assessing Language Testing: A Corpus-Based Approach

Grammar-based test suite construction using coverage-directed algorithms over LR-graphs

Automated Test Oracle Generation via Denotational Semantics

A System for Automated Unit Test Generation Using Large Language Models and Assessment of Generated Test Suites

SWT-Bench: Testing and Validating Real-World Bug-Fixes with Code Agents

Search-based Testing for Scratch Programs

Are We Testing or Being Tested? Exploring the Practical Applications of Large Language Models in Software Testing

Language testing

Adaptive Test Generation Using a Large Language Model

Software testing with code-based test generators: data and lessons learned from a case study with an industrial software component