Code datasets, often collected from diverse and uncontrolled sources such as
GitHub, potentially suffer from quality issues, thereby affecting the
performance and training efficiency of Large Language Models (LLMs) optimized
for code generation. Previous studies demonstrated the benefit of using
embedding spaces for data pruning, but they mainly focused on duplicate removal
or increasing variety, and in other modalities, such as images. Our work
focuses on using embeddings to identify and remove "low-quality" code data.
First, we explore features of "low-quality" code in embedding space, through
the use of synthetic corruptions. Armed with this knowledge, we devise novel
pruning metrics that operate in embedding space to identify and remove
low-quality entries in the Stack dataset. We demonstrate the benefits of this
synthetic corruption informed pruning (SCIP) approach on the well-established
HumanEval and MBPP benchmarks, outperforming existing embedding-based methods.
Importantly, we achieve up to a 3% performance improvement over no pruning,
thereby showing the promise of insights from synthetic corruptions for data
pruning.

利用嵌入空间检测和去除低质量代码数据的研究工作表明，采用合成污染信息的剪枝方法 (SCIP) 在数据清理方面取得了显著的性能提升，并展示了合成污染对数据剪枝的有益见解。

通过合成异常数据解码数据质量：基于嵌入式指导的代码数据剪枝

Decoding Data Quality via Synthetic Corruptions: Embedding-guided  Pruning of Code Data

Code datasets are of immense value for training neural-network-based code
completion models, where companies or organizations have made substantial
investments to establish and process these datasets. Unluckily, these datasets,
either built for proprietary or public usage, face the high risk of
unauthorized exploits, resulting from data leakages, license violations, etc.
Even worse, the ``black-box'' nature of neural models sets a high barrier for
externals to audit their training datasets, which further connives these
unauthorized usages. Currently, watermarking methods have been proposed to
prohibit inappropriate usage of image and natural language datasets. However,
due to domain specificity, they are not directly applicable to code datasets,
leaving the copyright protection of this emerging and important field of code
data still exposed to threats. To fill this gap, we propose a method, named
CodeMark, to embed user-defined imperceptible watermarks into code datasets to
trace their usage in training neural code completion models. CodeMark is based
on adaptive semantic-preserving transformations, which preserve the exact
functionality of the code data and keep the changes covert against
rule-breakers. We implement CodeMark in a toolkit and conduct an extensive
evaluation of code completion models. CodeMark is validated to fulfill all
desired properties of practical watermarks, including harmlessness to model
accuracy, verifiability, robustness, and imperceptibility.

应对代码数据集的版权保护问题，我们提出了一种名为 CodeMark 的方法，通过自适应的语义保持转换，将用户定义的隐形水印嵌入到代码数据集中，以追踪其在训练神经网络代码完成模型中的使用。CodeMark 工具包经过广泛评估，被验证具有实际水印所需的全部属性，包括对模型准确性无害、可验证性、鲁棒性和不可察觉性。

CodeMark：针对神经代码补全模型的隐形代码数据标记

CodeMark: Imperceptible Watermarking for Code Datasets against Neural  Code Completion Models

While large code datasets have become available in recent years, acquiring
representative training data with full coverage of general code distribution
remains challenging due to the compositional nature of code and the complexity
of software. This leads to the out-of-distribution (OOD) issues with unexpected
model inference behaviors that have not been systematically studied yet. We
contribute the first systematic approach that simulates various OOD scenarios
along different dimensions of data properties and investigates the model
behaviors in such scenarios. Our extensive studies on six state-of-the-art
models for three code generation tasks expose several failure modes caused by
the out-of-distribution issues. It thereby provides insights and sheds light
for future research in terms of generalization, robustness, and inductive
biases of source code models.

由于代码的组合性和软件的复杂性，获取具有完整代码分布覆盖的代表性训练数据仍然具有挑战性。因此，我们提出了第一种系统性方法，模拟不同维度数据属性的各种 OOD 场景，并研究在这些场景下模型的行为。我们对三项代码生成任务中的六种最先进的模型进行了广泛的研究，发现了一些由 OOD 问题引起的失败模式。此研究具有启示作用，为未来关于源代码模型的泛化、鲁棒性和归纳偏差等的研究提供了新的视角。