With the advent of neural language models, the performance of code generation has been significantly boosted. However, the problem of repetitions during the generation process continues to linger. Previous work has primarily focused on content repetition, which is merely a fraction of the broader repetition problem in code generation. A more prevalent and challenging problem is structural repetition. In structural repetition, the repeated code appears in various patterns but possesses a fixed structure, which can be inherently reflected in grammar. In this paper, we formally define structural repetition and propose an efficient decoding approach called RPG, which stands for Repetition Penalization based on Grammar, to alleviate the repetition problems in code generation for LLMs. Specifically, RPG first leverages grammar rules to identify repetition problems during code generation, and then strategically decays the likelihood of critical tokens that contribute to repetitions, thereby mitigating them in code generation. To facilitate this study, we construct a new dataset CodeRepetEval to comprehensively evaluate approaches for mitigating the repetition problems in code generation. Extensive experimental results demonstrate that RPG substantially outperforms the best-performing baselines on CodeRepetEval dataset as well as HumanEval and MBPP benchmarks, effectively reducing repetitions and enhancing the quality of generated code.

本研究解决了代码生成中存在的结构性重复问题，这是之前研究主要集中于内容重复之外的更广泛且具有挑战性的难题。提出了一种名为RPG的高效解码方法，通过利用语法规则识别重复问题，并逐步降低导致重复的关键令牌的概率，显著减少生成代码中的重复现象。实验结果表明，RPG在新构建的数据集CodeRepetEval及其它基准测试中明显优于现有方法，提升了生成代码的质量。

重新思考大型语言模型在代码生成中的重复问题