Large language models (LLMs) have shown remarkable performance on code
generation tasks. A recent application of LLMs for code generation is iterative
code repair, where a model fixes an incorrect program by rationalizing about
errors and generating a new program. However, code repair is primarily studied
on high-resource languages like Python, and the framework's efficacy is
under-explored on low-resource languages. To apply code repair for low-resource
languages, we propose Distilling Low-Resource Repairs (DistiLRR), an approach
that transfers the reasoning and code generation ability from a teacher model
to a student model. Our results show that DistiLRR consistently outperforms
baselines on low-resource languages, but has similar performance on
high-resource languages. To investigate this behavior, we perform a further
analysis and find that the correlation between rationale quality and code
correctness is weaker than previously perceived. We hypothesize this weakness
is magnified in low-resource settings where base models lack deep knowledge of
a programming language, leading to wavering benefits of code repair between
high-resource and low-resource languages.

使用大型语言模型（LLMs）进行代码修复可以在高资源语言和低资源语言上实现较好的性能表现，但低资源语言中的代码修复效果受到了深知识缺乏的基础模型的限制。我们提出了一种名为 DistiLRR 的方法，通过从教师模型到学生模型的推理和代码生成能力转移，可以在低资源语言中获得比基线模型更好的性能。研究结果显示 DistiLRR 在低资源语言上始终优于基线模型，但在高资源语言上性能相似。我们经过进一步分析，发现了解释质量与代码正确性之间的相关性较弱，这种弱点在低资源环境中被放大，因为基础模型缺乏对编程语言的深入知识，导致代码修复在高资源和低资源语言之间的效果存在差异。

DistiLRR：低资源编程语言的代码修复迁移

DistiLRR: Transferring Code Repair for Low-Resource Programming  Languages

Two main approaches for evaluating the quality of machine-generated
rationales are: 1) using human rationales as a gold standard; and 2) automated
metrics based on how rationales affect model behavior. An open question,
however, is how human rationales fare with these automatic metrics. Analyzing a
variety of datasets and models, we find that human rationales do not
necessarily perform well on these metrics. To unpack this finding, we propose
improved metrics to account for model-dependent baseline performance. We then
propose two methods to further characterize rationale quality, one based on
model retraining and one on using "fidelity curves" to reveal properties such
as irrelevance and redundancy. Our work leads to actionable suggestions for
evaluating and characterizing rationales.

本研究提出了两种度量机器生成的理由质量的方法，分别是使用人工理由作为金标准和基于理由对模型行为的影响的自动化指标。然而，人工理由与这些自动化指标的匹配程度仍存在问题。我们提出了改进的指标来解决这个问题，并提出了两种方法来进一步表征理由质量，一种是基于模型再训练，另一种是使用 “保真度曲线” 来揭示诸如无关性和冗余性等特性。我们的研究提出了实际的建议，以评估和表征理由。