Large language models trained on massive amounts of natural language data and
code have shown impressive capabilities in automatic code generation scenarios.
Development and evaluation of these models has largely been driven by offline
functional correctness metrics, which consider a task to be solved if the
generated code passes corresponding unit tests. While functional correctness is
clearly an important property of a code generation model, we argue that it may
not fully capture what programmers value when collaborating with their AI pair
programmers. For example, while a nearly correct suggestion that does not
consider edge cases may fail a unit test, it may still provide a substantial
starting point or hint to the programmer, thereby reducing total needed effort
to complete a coding task. To investigate this, we conduct a user study with
(N=49) experienced programmers, and find that while both correctness and effort
correlate with value, the association is strongest for effort. We argue that
effort should be considered as an important dimension of evaluation in code
generation scenarios.
We also find that functional correctness remains better at identifying the
highest-value generations; but participants still saw considerable value in
code that failed unit tests. Conversely, similarity-based metrics are very good
at identifying the lowest-value generations among those that fail unit tests.
Based on these findings, we propose a simple hybrid metric, which combines
functional correctness and similarity-based metrics to capture different
dimensions of what programmers might value and show that this hybrid metric
more strongly correlates with both value and effort. Our findings emphasize the
importance of designing human-centered metrics that capture what programmers
need from and value in their AI pair programmers.

研究表明，具有实现代码自动生成能力的大型语言模型的正确性及努力度均对程序员的价值有影响，建议设计出更人性化的评估指标以评估这些模型的优劣。

将人工智能配对编程的离线度量与人类价值判断对齐

Aligning Offline Metrics and Human Judgments of Value of AI-Pair Programmers

Sequence-processing neural networks led to remarkable progress on many NLP
tasks. As a consequence, there has been increasing interest in understanding to
what extent they process language as humans do. We aim here to uncover which
biases such models display with respect to "natural" word-order constraints. We
train models to communicate about paths in a simple gridworld, using miniature
languages that reflect or violate various natural language trends, such as the
tendency to avoid redundancy or to minimize long-distance dependencies. We
study how the controlled characteristics of our miniature languages affect
individual learning and their stability across multiple network generations.
The results draw a mixed picture. On the one hand, neural networks show a
strong tendency to avoid long-distance dependencies. On the other hand, there
is no clear preference for the efficient, non-redundant encoding of information
that is widely attested in natural language. We thus suggest inoculating a
notion of "effort" into neural networks, as a possible way to make their
linguistic behavior more human-like.

本研究旨在发现序列处理神经网络对于 “自然” 词序约束的偏见。结果表明，神经网络倾向于避免长距离依赖，但并没有明显的偏好于高效的，非冗余的信息编码。因此我们建议在神经网络中引入 “努力程度” 的概念，以使其语言行为更像人类。