Large Language Models (LLMs) have demonstrated potential in assisting with
Register Transfer Level (RTL) design tasks. Nevertheless, there remains to be a
significant gap in benchmarks that accurately reflect the complexity of
real-world RTL projects. To address this, this paper presents RTL-Repo, a
benchmark specifically designed to evaluate LLMs on large-scale RTL design
projects. RTL-Repo includes a comprehensive dataset of more than 4000 Verilog
code samples extracted from public GitHub repositories, with each sample
providing the full context of the corresponding repository. We evaluate several
state-of-the-art models on the RTL-Repo benchmark, including GPT-4, GPT-3.5,
Starcoder2, alongside Verilog-specific models like VeriGen and RTLCoder, and
compare their performance in generating Verilog code for complex projects. The
RTL-Repo benchmark provides a valuable resource for the hardware design
community to assess and compare LLMs' performance in real-world RTL design
scenarios and train LLMs specifically for Verilog code generation in complex,
multi-file RTL projects. RTL-Repo is open-source and publicly available on
Github.

该研究提出了 RTL-Repo 基准，并通过评估多个先进模型在该基准上的表现，比较了它们在生成复杂 RTL 项目的 Verilog 代码方面的性能。RTL-Repo 基准为硬件设计社区提供了一个有价值的资源，用于评估和比较大型语言模型在真实 RTL 设计场景中的性能，并专门为复杂的多文件 RTL 项目的 Verilog 代码生成训练大型语言模型。RTL-Repo 是一个开源项目，可在 Github 上公开获取。