Abstract:In digital circuit design, testbenches constitute the cornerstone of simulation-based hardware verification. Traditional methodologies for testbench generation during simulation-based hardware verification still remain partially manual, resulting in inefficiencies in testing various scenarios and requiring expensive time from designers. Large Language Models (LLMs) have demonstrated their potential in automating the circuit design flow. However, directly applying LLMs to generate testbenches suffers from a low pass rate. To address this challenge, we introduce AutoBench, the first LLM-based testbench generator for digital circuit design, which requires only the description of the design under test (DUT) to automatically generate comprehensive testbenches. In AutoBench, a hybrid testbench structure and a self-checking system are realized using LLMs. To validate the generated testbenches, we also introduce an automated testbench evaluation framework to evaluate the quality of generated testbenches from multiple perspectives. Experimental results demonstrate that AutoBench achieves a 57% improvement in the testbench pass@1 ratio compared with the baseline that directly generates testbenches using LLMs. For 75 sequential circuits, AutoBench successfully has a 3.36 times testbench pass@1 ratio compared with the baseline. The source codes and experimental results are open-sourced at this link: <a class="link-external link-https" href="https://github.com/AutoBench/AutoBench" rel="external noopener nofollow">this https URL</a>

What problem does this paper attempt to address?

### What problems does this paper attempt to solve? This paper aims to solve the challenges in the generation of automatic testbenches during the hardware verification process. Specifically, traditional testbench generation methods still rely partly on manual operations in analog hardware verification, resulting in low efficiency and long time - consuming. Although large - language models (LLMs) have shown potential in the automated circuit design flow, the direct application of LLMs to generate testbenches is not ideal, with a low pass rate. To solve these problems, the paper proposes **AutoBench** - an LLM - based automatic testbench generation and evaluation framework. The main objectives of AutoBench are: 1. **Automatically generate comprehensive testbenches**: Only by providing the description of the design - under - test (DUT), a hybrid - structure testbench containing drivers and checkers can be automatically generated. 2. **Improve the quality of testbenches**: By introducing an automatic evaluation framework (AutoEval), the quality of the generated testbenches is evaluated from multiple perspectives to ensure its effectiveness and reliability. 3. **Achieve full automation**: Not only generate test vectors, but also automatically check the signals of the DUT, thus realizing the automation of the entire simulation verification process. Experimental results show that, compared with directly using LLMs to generate testbenches, AutoBench has a significant improvement in the testbench pass rate. For 75 sequential circuits, the pass rate of AutoBench is increased by 3.36 times. ### Main contributions - Proposed the first systematic and general Verilog testbench generation method based on LLMs and verified it on a large - scale dataset. - Designed a hybrid testbench architecture that combines Python code generated by LLM, Verilog code and Python scripts. - Introduced a comprehensive set of LLM code generation methods, including hybrid code generation, scenario checking, code standardization and LLM automatic debugging. - Proposed an automatic evaluation framework AutoEval for evaluating the quality of the generated testbenches and extended the existing dataset to support a broader evaluation. Through these contributions, AutoBench provides a powerful tool for automated hardware verification, significantly improving the efficiency and quality of testbench generation.

AutoBench: Automatic Testbench Generation and Evaluation Using LLMs for HDL Design

CorrectBench: Automatic Testbench Generation with Functional Self-Correction using LLMs for HDL Design

Accelerating RTL Quality Assessment Method Based on CAD Tools

Design of automatic hydraulic test bench based on LabVIEW

LLM-Aided Testbench Generation and Bug Detection for Finite-State Machines

AssertionBench: A Benchmark to Evaluate Large-Language Models for Assertion Generation

AutoChip: Automating HDL Generation Using LLM Feedback

Towards LLM-Powered Verilog RTL Assistant: Self-Verification and Self-Correction

Evaluating LLMs for Hardware Design and Test

LLM-Aided Efficient Hardware Design Automation

Benchmarking Large Language Models for Automated Verilog RTL Code Generation

ChiBench: a Benchmark Suite for Testing Electronic Design Automation Tools

VerilogReader: LLM-Aided Hardware Test Generation

Can EDA Tool Feedback Improve Verilog Generation by LLMs?

A Verification Methodology for Reusable Test Cases and Coverage Based on System Verilog

AnalogGym: An Open and Practical Testing Suite for Analog Circuit Synthesis

Classification-Based Automatic HDL Code Generation Using LLMs

BetterV: Controlled Verilog Generation with Discriminative Guidance

LLM4DV: Using Large Language Models for Hardware Test Stimuli Generation

BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions

AssertLLM: Generating and Evaluating Hardware Verification Assertions from Design Specifications via Multi-LLMs