We present The Vault, an open-source, large-scale code-text dataset designed
to enhance the training of code-focused large language models (LLMs). Existing
open-source datasets for training code-based LLMs often face challenges in
terms of size, quality (due to noisy signals), and format (only containing code
function and text explanation pairings). The Vault overcomes these limitations
by providing 40 million code-text pairs across 10 popular programming
languages, thorough cleaning for 10+ prevalent issues, and various levels of
code-text pairings, including class, function, and line levels. Researchers and
practitioners can utilize The Vault for training diverse code-focused LLMs or
incorporate the provided data cleaning methods and scripts to improve their
datasets. By employing The Vault as the training dataset for code-centric LLMs,
we anticipate significant advancements in code understanding and generation
tasks, fostering progress in both artificial intelligence research and software
development practices.

The Vault 是一个开源的、大规模的代码文本数据集，具备 10 种流行编程语言的 4000 万个代码文本对，经过清理和处理，可用于训练不同类型的基于代码的大型语言模型，并同时为研究者和实践者提供了数据清理方法和脚本以改善他们的数据集。

Vault：一个全面的多语言数据集，用于推进代码理解和生成

The Vault: A Comprehensive Multilingual Dataset for Advancing Code  Understanding and Generation

Existing models on Machine Reading Comprehension (MRC) require complex model
architecture for effectively modeling long texts with paragraph representation
and classification, thereby making inference computationally inefficient for
production use. In this work, we propose VAULT: a light-weight and
parallel-efficient paragraph representation for MRC based on contextualized
representation from long document input, trained using a new Gaussian
distribution-based objective that pays close attention to the partially correct
instances that are close to the ground-truth. We validate our VAULT
architecture showing experimental results on two benchmark MRC datasets that
require long context modeling; one Wikipedia-based (Natural Questions (NQ)) and
the other on TechNotes (TechQA). VAULT can achieve comparable performance on NQ
with a state-of-the-art (SOTA) complex document modeling approach while being
16 times faster, demonstrating the efficiency of our proposed model. We also
demonstrate that our model can also be effectively adapted to a completely
different domain -- TechQA -- with large improvement over a model fine-tuned on
a previously published large PLM.

本论文提出了一种基于上下文表示的轻量且并行高效的 MRC 段落表示方法 VAULT，通过使用一种新的基于高斯分布的目标函数训练模型，其关注部分正确的实例的方法，从而克服了现有模型需要复杂模型架构的瓶颈，实验结果表明 VAULT 在处理长文本时比 SOTA 的复杂文档建模方法更快 16 倍，在 NQ 数据集上的性能相当。同时，本模型在不同领域上的有效性也得到验证。