Code datasets, often collected from diverse and uncontrolled sources such as
GitHub, potentially suffer from quality issues, thereby affecting the
performance and training efficiency of Large Language Models (LLMs) optimized
for code generation. Previous studies demonstrated the benefit of using
embedding spaces for data pruning, but they mainly focused on duplicate removal
or increasing variety, and in other modalities, such as images. Our work
focuses on using embeddings to identify and remove "low-quality" code data.
First, we explore features of "low-quality" code in embedding space, through
the use of synthetic corruptions. Armed with this knowledge, we devise novel
pruning metrics that operate in embedding space to identify and remove
low-quality entries in the Stack dataset. We demonstrate the benefits of this
synthetic corruption informed pruning (SCIP) approach on the well-established
HumanEval and MBPP benchmarks, outperforming existing embedding-based methods.
Importantly, we achieve up to a 3% performance improvement over no pruning,
thereby showing the promise of insights from synthetic corruptions for data
pruning.

利用嵌入空间检测和去除低质量代码数据的研究工作表明，采用合成污染信息的剪枝方法 (SCIP) 在数据清理方面取得了显著的性能提升，并展示了合成污染对数据剪枝的有益见解。