Abstract:High-Level Synthesis (HLS) tools convert C/C++ design code into Hardware Description Language (HDL) code automatically, which are often used for Field Programmable Gate Array (FPGA) design. HLS tools provide many pragmas, which are a kind of directive to be inserted into C/C++ code, for designers to efficiently control the synthesis of code components (e.g., arrays and loops) to generate FPGA implementations with varying performances and costs. However, the use of some pragmas may trigger HLS tool bugs (e.g., tool crashes). Although many formal methods have been proposed to verify the correctness of various HLS phases, no relevant work addresses the problem on detecting HLS tool pragma-related bugs. To resolve this problem, two challenges need to be addressed, namely the selection of testing programs and the acquisition of pragma combinations, due to the enormous number of testing programs and pragma combinations. In this paper, we propose TEPACS, a TEsting Program and prAgma Combination Selection-based framework, to construct diverse testing programs with pragmas for effectively detecting HLS tool pragma-related bugs. TEPACS follows the idea of fuzzing, which is a widely used technique in software testing. First, TEPACS selects the representative testing program according to the cosine distance between the code component vectors of testing programs. Then, for a selected program, TEPACS generates its golden output and uses the pragma combination selection method based on combinatorial testing to generate a set of programs with different pragmas. TEPACS uses the HLS tool under test to convert these testing programs into HDL codes and obtains the simulation results of the HDL code. Finally, based on differential testing, TEPACS identifies HLS tool bugs triggered if the simulation result and golden output are inconsistent. We evaluate TEPACS and its five variants on Vitis HLS, a widely used FPGA HLS tool. Experimental results show that TEPACS outperforms the baselines by at least 11.17% in terms of the bug-finding capability. In one month, TEPACS detected 34 bugs on the latest version of Vitis HLS, of which 9 bugs have been confirmed.

Mantra: Mutation Testing of Hardware Design Code Based on Real Bugs.

MorFuzz: Fuzzing Processor Via Runtime Instruction Morphing Enhanced Synchronizable Co-simulation.

Knowledge-Augmented Mutation-Based Bug Localization for Hardware Design Code

Mutation-based diagnostic test generation for hardware design error diagnosis

MOTIF: A tool for Mutation Testing with Fuzzing

Functional Testbench Qualification by Mutation Analysis

Mutation Testing: Principal, Optimization and Application

Time-Aware Spectrum-Based Bug Localization for Hardware Design Code with Data Purification

Mutation Testing of Memory-Related Operators

Mutation-analysis Driven Functional Verification of a Soft Microprocessor.

The State-of-the-art in EDA tools: the Hardware Trojan Confrontation Framework

Fuzzing for CPS Mutation Testing

Mutation Testing in Practice: Insights from Open-Source Software Developers

Automatic Debugging of Operator Errors Based on Efficient Mutation Analysis

Specification-Driven Conformance Checking for Virtual/Silicon Devices Using Mutation Testing

Heuristics for Strongly Killing Mutants in Software Mutation Testing

A Testing Program and Pragma Combination Selection Based Framework for High-Level Synthesis Tool Pragma-Related Bug Detection

Which Quantum Circuit Mutants Shall Be Used? An Empirical Evaluation of Quantum Circuit Mutations

Test Data Generation for Mutation Testing Based on Markov Chain Usage Model and Estimation of Distribution Algorithm

Grammar-based fuzz testing for microprocessor RTL design

Efficient Mutation-Analysis Coverage For Constrained Random Verification