Abstract:Assertions have been the de facto collateral for simulation-based and formal verification of hardware designs for over a decade. The quality of hardware verification, \ie, detection and diagnosis of corner-case design bugs, is critically dependent on the quality of the assertions. There has been a considerable amount of research leveraging a blend of data-driven statistical analysis and static analysis to generate high-quality assertions from hardware design source code and design execution trace data. Despite such concerted effort, all prior research struggles to scale to industrial-scale large designs, generates too many low-quality assertions, often fails to capture subtle and non-trivial design functionality, and does not produce any easy-to-comprehend explanations of the generated assertions to understand assertions' suitability to different downstream validation tasks. Recently, with the advent of Large-Language Models (LLMs), there has been a widespread effort to leverage prompt engineering to generate assertions. However, there is little effort to quantitatively establish the effectiveness and suitability of various LLMs for assertion generation. In this paper, we present AssertionBench, a novel benchmark to evaluate LLMs' effectiveness for assertion generation quantitatively. AssertioBench contains 100 curated Verilog hardware designs from OpenCores and formally verified assertions for each design generated from GoldMine and HARM. We use AssertionBench to compare state-of-the-art LLMs to assess their effectiveness in inferring functionally correct assertions for hardware designs. Our experiments demonstrate how LLMs perform relative to each other, the benefits of using more in-context exemplars in generating a higher fraction of functionally correct assertions, and the significant room for improvement for LLM-based assertion generators.

An SVA Hardware Monitor with Off-Line Replay

Application of SVA in Hardware functional verification

A Generic UVM Verification Framework Based on Two-Stage Comparison for Protocol-Agnostic Data Streams in Digital Designs

An innovative embedded logic analyzer based SoC verification platform

Online Model Checking for Dependable Real-Time Systems

LAAG-RV: LLM Assisted Assertion Generation for RTL Design Verification

Software/hareware hybrid monitoring technique in embedded system design

Automatic High-quality Verilog Assertion Generation through Subtask-Focused Fine-Tuned LLMs and Iterative Prompting

A System-Level SOC Verification Method Based on Hardware Accelerator

Rabbit: an Efficient Verification Platform Base on Virtual Peripherals

A Hardware-Software Cooperative Interval-Replaying for FPGA-based Architecture Evaluation

Track Down HW Function Faults Using Real SW Invariants

VERIS: an Efficient Model Checker for Synchronous VHDL Designs

Leveraging Hardware-Assisted Virtualization for Deterministic Replay on Commodity Multi-Core Processors

ASAX: Automatic Security Assertion Extraction for Detecting Hardware Trojans

AssertionBench: A Benchmark to Evaluate Large-Language Models for Assertion Generation

Hardware/Software Co-verification Using Path-based Symbolic Execution

Virtual-Peripheral-in-the-Loop : A Hardware-in-the-Loop Strategy to Bridge the VP/RTL Design-Gap

V2Va: An Efficient Verilog-to-Verilog-A Translator for Accelerated Mixed-Signal Simulation

FVEval: Understanding Language Model Capabilities in Formal Verification of Digital Hardware

Formal Specification-based Online Monitoring