The increasing popularity of large language models (LLMs) has paved the way
for their application in diverse domains. This paper proposes a benchmarking
framework tailored specifically for evaluating LLM performance in the context
of Verilog code generation for hardware design and verification. We present a
comprehensive evaluation dataset consisting of 156 problems from the Verilog
instructional website HDLBits. The evaluation set consists of a diverse set of
Verilog code generation tasks, ranging from simple combinational circuits to
complex finite state machines. The Verilog code completions can be
automatically tested for functional correctness by comparing the transient
simulation outputs of the generated design with a golden solution. We also
demonstrate that the Verilog code generation capability of pretrained language
models could be improved with supervised fine-tuning by bootstrapping with LLM
generated synthetic problem-code pairs.

该论文提出了一种专门为评估大型语言模型在硬件设计和验证中的 Verilog 代码生成性能而设计的基准测试框架，演示了预训练语言模型的 Verilog 代码生成能力可以通过使用 LLM 生成的合成问题 - 代码对进行监督微调来提高。