Large Language Models' success on text generation has also made them better
at code generation and coding tasks. While a lot of work has demonstrated their
remarkable performance on tasks such as code completion and editing, it is
still unclear as to why. We help bridge this gap by exploring to what degree
auto-regressive models understand the logical constructs of the underlying
programs. We propose Counterfactual Analysis for Programming Concept Predicates
(CACP) as a counterfactual testing framework to evaluate whether Large Code
Models understand programming concepts. With only black-box access to the
model, we use CACP to evaluate ten popular Large Code Models for four different
programming concepts. Our findings suggest that current models lack
understanding of concepts such as data flow and control flow.

大型语言模型在文本生成方面的成功也使它们在代码生成和编码任务方面表现更好。然而，尽管有许多研究展示了它们在代码完成和编辑等任务上的出色性能，但是为什么如此仍然不清楚。我们通过研究来探讨自回归模型理解底层程序的逻辑结构的程度。我们提出了用于程序概念预测的反事实分析（Counterfactual Analysis for Programming Concept Predicates，CACP）作为一个反事实测试框架，以评估大型代码模型对编程概念的理解程度。我们通过只具有黑盒访问模型的能力，使用 CACP 评估了十个流行的大型代码模型在四个不同的编程概念上的表现。我们的研究结果表明当前的模型在数据流和控制流等概念上缺乏理解。

大型代码模型是否理解编程概念？黑盒方法

Do Large Code Models Understand Programming Concepts? A Black-box  Approach

Motivated by the increasing popularity of code generation from human
descriptions using large language models (LLMs), several benchmarks have been
proposed to assess the capabilities of existing and emerging models. This study
presents a large-scale human evaluation of HumanEval and MBPP, two widely used
benchmarks for Python code generation, focusing on their diversity and
difficulty. Our findings reveal a significant bias towards a limited number of
programming concepts, with negligible or no representation of most concepts.
Additionally, we identify a concerningly high proportion of easy programming
questions, potentially leading to an overestimation of model performance on
code generation tasks.

本研究对 Python 代码生成的两个广泛应用的基准测试 ——HumanEval 和 MBPP 进行了大规模人工评估，重点关注它们的多样性与难度。研究结果发现，现有基准测试对少数编程概念存在显著偏向，而对大多数概念几乎没有或没有代表性的呈现。此外，研究还发现易于完成的编程问题所占比例极高，可能导致对模型在代码生成任务上性能的过高评价。

挑战前人未达之地：暴露代码生成评估中的偏见和不足

Boldly Going Where No Benchmark Has Gone Before: Exposing Bias and  Shortcomings in Code Generation Evaluation

A multitude of different probabilistic programming languages exists today,
all extending a traditional programming language with primitives to support
modeling of complex, structured probability distributions. Each of these
languages employs its own probabilistic primitives, and comes with a particular
syntax, semantics and inference procedure. This makes it hard to understand the
underlying programming concepts and appreciate the differences between the
different languages. To obtain a better understanding of probabilistic
programming, we identify a number of core programming concepts underlying the
primitives used by various probabilistic languages, discuss the execution
mechanisms that they require and use these to position state-of-the-art
probabilistic languages and their implementation. While doing so, we focus on
probabilistic extensions of logic programming languages such as Prolog, which
have been developed since more than 20 years.

本文旨在通过探讨各种不同的概率编程语言所使用的基本概率原语，包括逻辑编程语言的概率扩展，从而比较和理解它们之间的区别，以更好地理解概率编程的核心编程概念和执行机制。