ITERTL: An Iterative Framework for Fine-tuning LLMs for RTL Code Generation

Peiyang Wu,Nan Guo,Xiao Xiao,Wenming Li,Xiaochun Ye,Dongrui Fan

2024-06-28

Abstract:Recently, large language models (LLMs) have demonstrated excellent performance in understanding human instructions and generating code, which has inspired researchers to explore the feasibility of generating RTL code with LLMs. However, the existing approaches to fine-tune LLMs on RTL codes typically are conducted on fixed datasets, which do not fully stimulate the capability of LLMs and require large amounts of reference data. To mitigate these issues , we introduce a simple yet effective iterative training paradigm named ITERTL. During each iteration, samples are drawn from the model trained in the previous cycle. Then these new samples are employed for training in this loop. Through this iterative approach, the distribution mismatch between the model and the training samples is reduced. Additionally, the model is thus enabled to explore a broader generative space and receive more comprehensive feedback. Theoretical analyses are conducted to investigate the mechanism of the effectiveness. Experimental results show the model trained through our proposed approach can compete with and even outperform the state-of-the-art (SOTA) open-source model with nearly 37\% reference samples, achieving remarkable 42.9\% and 62.2\% pass@1 rate on two VerilogEval evaluation datasets respectively. While using the same amount of reference samples, our method can achieved a relative improvement of 16.9\% and 12.5\% in pass@1 compared to the non-iterative method. This study facilitates the application of LLMs for generating RTL code in practical scenarios with limited data.

Computation and Language,Artificial Intelligence

What problem does this paper attempt to address?

The problem this paper attempts to address is that existing large language models (LLMs) face two main issues when generating RTL code:

1. **Limited exploration space**: Existing methods typically fine-tune LLMs on a fixed dataset, which limits the model's exploration range and results in incomplete feedback signals.
2. **Distribution mismatch**: There is a mismatch between the distribution of training samples and the distribution of the LLM being trained, which can lead to estimation errors during the optimization process.

To address these issues, the authors propose an iterative training framework (ITERTL). By extracting samples from the previously trained model in each iteration and training with these new samples, the framework reduces distribution mismatch, expands the model's exploration space, and obtains more comprehensive feedback. Experimental results show that this method can achieve or even surpass the performance of existing state-of-the-art (SOTA) open-source models with fewer reference samples.