Large Language Models are becoming an increasingly popular tool for software
development. Their ability to model and generate source code has been
demonstrated in a variety of contexts, including code completion,
summarization, translation, and lookup. However, they often struggle to
generate code for more complex tasks. In this paper, we explore the ability of
state-of-the-art language models to generate parallel code. We propose a
benchmark, PCGBench, consisting of a set of 420 tasks for evaluating the
ability of language models to generate parallel code, and we evaluate the
performance of several state-of-the-art open- and closed-source language models
on these tasks. We introduce novel metrics for comparing parallel code
generation performance and use them to explore how well each LLM performs on
various parallel programming models and computational problem types.

本研究提出了一个用于评估最新语言模型在生成并行代码方面能力的基准测试集，以及评估了多个开源和闭源语言模型在此基准测试集上的性能，并引入了用于比较并行代码生成性能的新指标，旨在探讨每个语言模型在不同并行编程模型和计算问题类型上的表现。

大型语言模型能否编写并行代码？

Can Large Language Models Write Parallel Code?

With easier access to powerful compute resources, there is a growing trend in
AI for software development to develop larger language models (LLMs) to address
a variety of programming tasks. Even LLMs applied to tasks from the
high-performance computing (HPC) domain are huge in size and demand expensive
compute resources for training. This is partly because these LLMs for HPC tasks
are obtained by finetuning existing LLMs that support several natural and/or
programming languages. We found this design choice confusing - why do we need
large LMs trained on natural languages and programming languages unrelated to
HPC for HPC-specific tasks?
In this line of work, we aim to question choices made by existing LLMs by
developing smaller LMs for specific domains - we call them domain-specific LMs.
Specifically, we start off with HPC as a domain and build an HPC-specific LM,
named MonoCoder, that is orders of magnitude smaller than existing LMs but
delivers similar, if not better performance, on non-HPC and HPC tasks.
Specifically, we pre-trained MonoCoder on an HPC-specific dataset (named
HPCorpus) of C and C++ programs mined from GitHub. We evaluated the performance
of MonoCoder against conventional multi-lingual LLMs. Results demonstrate that
MonoCoder, although much smaller than existing LMs, achieves similar results on
normalized-perplexity tests and much better ones in CodeBLEU competence for
high-performance and parallel code generations. Furthermore, fine-tuning the
base model for the specific task of parallel code generation (OpenMP parallel
for pragmas) demonstrates outstanding results compared to GPT, especially when
local misleading semantics are removed by our novel pre-processor Tokompiler,
showcasing the ability of domain-specific models to assist in HPC-relevant
tasks.

通过开发针对特定领域的较小语言模型，我们质疑现有的语言模型选择，构建了一个针对高性能计算（HPC）领域的 MonoCoder 模型，它在非 HPC 和 HPC 任务中实现了类似甚至更好的性能，并在高性能和并行代码生成中取得出色结果。