Input-Gen: Guided Generation of Stateful Inputs for Testing, Tuning, and Training

Ivan R. Ivanov,Joachim Meyer,Aiden Grossman,William S. Moses,Johannes Doerfert

2024-06-13

Abstract:The size and complexity of software applications is increasing at an accelerating pace. Source code repositories (along with their dependencies) require vast amounts of labor to keep them tested, maintained, and up to date. As the discipline now begins to also incorporate automatically generated programs, automation in testing and tuning is required to keep up with the pace - let alone reduce the present level of complexity. While machine learning has been used to understand and generate code in various contexts, machine learning models themselves are trained almost exclusively on static code without inputs, traces, or other execution time information. This lack of training data limits the ability of these models to understand real-world problems in software. In this work we show that inputs, like code, can be generated automatically at scale. Our generated inputs are stateful, and appear to faithfully reproduce the arbitrary data structures and system calls required to rerun a program function. By building our tool within the compiler, it both can be applied to arbitrary programming languages and architectures and can leverage static analysis and transformations for improved performance. Our approach is able to produce valid inputs, including initial memory states, for 90% of the ComPile dataset modules we explored, for a total of 21.4 million executable functions. Further, we find that a single generated input results in an average block coverage of 37%, whereas guided generation of five inputs improves it to 45%.

Software Engineering,Performance,Programming Languages

What problem does this paper attempt to address?

The paper aims to address the problem of automatically generating stateful inputs during software testing, tuning, and training processes. Specifically, the authors propose a method that can automatically generate inputs, including memory states, for arbitrary program fragments without requiring any user-provided inputs. This method is implemented through a compiler plugin, which can be applied to various programming languages and architectures, and leverages static analysis and transformation to improve performance.

The main contributions include:
1. Proposing a framework that can create stateful inputs for any programming language compiled to LLVM.
2. Providing static and dynamic guidance to improve branch coverage. Through hints and branch profiling provided by the compiler, the runtime will select values that guide the control flow to unexplored areas.
3. Conducting a case study on matrix multiplication to explore the time and memory impact of input generation and replay.
4. Conducting research on the large-scale dataset ComPile, demonstrating the framework's performance on large-scale functions. Among 23.9 million functions, 99% could be instrumented, 90% could be successfully replayed, and the initial branch coverage increased from 37% to 45% (generating five inputs).