Rome was Not Built in a Single Step: Hierarchical Prompting for LLM-based Chip Design

Andre Nakkab,Sai Qian Zhang,Ramesh Karri,Siddharth Garg

2024-09-10

Abstract:Large Language Models (LLMs) are effective in computer hardware synthesis via hardware description language (HDL) generation. However, LLM-assisted approaches for HDL generation struggle when handling complex tasks. We introduce a suite of hierarchical prompting techniques which facilitate efficient stepwise design methods, and develop a generalizable automation pipeline for the process. To evaluate these techniques, we present a benchmark set of hardware designs which have solutions with or without architectural hierarchy. Using these benchmarks, we compare various open-source and proprietary LLMs, including our own fine-tuned Code Llama-Verilog model. Our hierarchical methods automatically produce successful designs for complex hardware modules that standard flat prompting methods cannot achieve, allowing smaller open-source LLMs to compete with large proprietary models. Hierarchical prompting reduces HDL generation time and yields savings on LLM costs. Our experiments detail which LLMs are capable of which applications, and how to apply hierarchical methods in various modes. We explore case studies of generating complex cores using automatic scripted hierarchical prompts, including the first-ever LLM-designed processor with no human feedback. Tools for the Recurrent Optimization via Machine Editing (ROME) method can be found at <a class="link-external link-https" href="https://github.com/ajn313/ROME-LLM" rel="external noopener nofollow">this https URL</a>

Hardware Architecture,Artificial Intelligence

What problem does this paper attempt to address?

### The Problem the Paper Attempts to Solve

The paper attempts to address the challenges of handling complex tasks when using large language models (LLMs) to generate hardware description language (HDL) code. Specifically, existing LLM methods perform poorly when dealing with complex hardware designs, especially when generating long and complex straight-line code, which often results in errors or irrelevant content. These issues lead to increased response delays and output length limitations.

To overcome these challenges, the authors introduce a set of hierarchical prompting techniques aimed at improving the efficiency and accuracy of LLMs in generating modular HDL code through a step-by-step design approach. These techniques include:

1. **Hierarchical Design**: Breaking down complex hardware modules into manageable submodules, similar to function definitions in high-level code.
2. **Automated Pipeline**: Developing a general automated pipeline to implement the hierarchical HDL generation process.
3. **Benchmarking**: Introducing a series of hardware design benchmarks, with or without architectural hierarchy, to evaluate the performance of different open-source and proprietary LLMs.
4. **Model Optimization**: Fine-tuning the Code Llama-Verilog model to make it more effective in generating HDL code.

Through these techniques, the authors aim to enable small open-source LLMs to compete with large proprietary models and reduce HDL generation time and LLM costs. Experimental results show that hierarchical prompting techniques significantly improve LLM performance in complex hardware design tasks, particularly on weaker models. Additionally, the technique successfully generated some complex hardware cores, including the first processor fully designed by an LLM without human feedback.