The programming skill is one crucial ability for Large Language Models
(LLMs), necessitating a deep understanding of programming languages (PLs) and
their correlation with natural languages (NLs). We examine the impact of
pre-training data on code-focused LLMs' performance by assessing the comment
density as a measure of PL-NL alignment. Given the scarcity of code-comment
aligned data in pre-training corpora, we introduce a novel data augmentation
method that generates comments for existing code, coupled with a data filtering
strategy that filters out code data poorly correlated with natural language. We
conducted experiments on three code-focused LLMs and observed consistent
improvements in performance on two widely-used programming skill benchmarks.
Notably, the model trained on the augmented data outperformed both the model
used for generating comments and the model further trained on the data without
augmentation.

我们研究了大型语言模型的编程技能对其性能的影响，并引入了一种新的数据增强方法和筛选策略来提高预训练数据在代码相关的语言模型性能上的表现。实验证明，使用增强数据训练的模型在两个广泛使用的编程技能评估上的表现优于生成评论的模型和没有使用增强数据进行训练的模型。

代码需要注释：用注释增强代码 LLMs

Code Needs Comments: Enhancing Code LLMs with Comment Augmentation

We present The Vault, an open-source, large-scale code-text dataset designed
to enhance the training of code-focused large language models (LLMs). Existing
open-source datasets for training code-based LLMs often face challenges in
terms of size, quality (due to noisy signals), and format (only containing code
function and text explanation pairings). The Vault overcomes these limitations
by providing 40 million code-text pairs across 10 popular programming
languages, thorough cleaning for 10+ prevalent issues, and various levels of
code-text pairings, including class, function, and line levels. Researchers and
practitioners can utilize The Vault for training diverse code-focused LLMs or
incorporate the provided data cleaning methods and scripts to improve their
datasets. By employing The Vault as the training dataset for code-centric LLMs,
we anticipate significant advancements in code understanding and generation
tasks, fostering progress in both artificial intelligence research and software
development practices.

The Vault 是一个开源的、大规模的代码文本数据集，具备 10 种流行编程语言的 4000 万个代码文本对，经过清理和处理，可用于训练不同类型的基于代码的大型语言模型，并同时为研究者和实践者提供了数据清理方法和脚本以改善他们的数据集。