The ability of CodeLLMs to generate executable and functionally correct code
at the \textit{repository-level scale }remains largely unexplored. We introduce
\methodnamews, a novel benchmark for evaluating code generation at the
repository-level scale, emphasizing executability and correctness.
\methodnamews provides an automated system that verifies requirements and
incorporates a mechanism for dynamically generating high-coverage test cases to
assess the functionality of generated code. Our work explores a controlled
scenario where developers specify necessary code dependencies, challenging the
model to integrate these accurately. Experiments show that while pretrained
LLMs outperform instruction-tuning models in correctness, the latter excel in
utilizing provided dependencies and demonstrating debugging capabilities.
\methodnamews aims to provide a comprehensive evaluation of code functionality
and alignment with developer intent, paving the way for more reliable and
applicable CodeLLMs in real-world scenarios.

CodeLLMs 在仓库级别规模上生成可执行且功能正确的代码的能力尚未得到广泛探索。我们引入了一种新的评估代码生成在仓库级别规模上的基准，名为 methodnamews，强调可执行性和正确性。methodnamews 提供了一个自动化系统，用于验证要求，并具有动态生成高覆盖率测试用例的机制，以评估生成代码的功能性。我们的研究探索了一个受控场景，开发人员在其中指定必要的代码依赖项，挑战模型准确地集成这些依赖项。实验证明，尽管预训练的 LLMs 在正确性方面胜过指令调优模型，但后者在利用所提供的依赖项和展示调试能力方面表现出色。methodnamews 旨在提供对代码功能和与开发人员意图的一致性的全面评估，为在实际场景中使用更可靠和适用的 CodeLLMs 铺平道路。

REPOEXEC: 通过代码库级别的可执行基准评估代码生成

REPOEXEC: Evaluate Code Generation with a Repository-Level Executable  Benchmark

We investigate Petri nets with data, an extension of plain Petri nets where
tokens carry values from an infinite data domain, and executability of
transitions is conditioned by equalities between data values. We provide a
decision procedure for the bi-reachability problem: given a Petri net and its
two configurations, we ask if each of the configurations is reachable from the
other. This pushes forward the decidability borderline, as the bi-reachability
problem subsumes the coverability problem (which is known to be decidable) and
is subsumed by the reachability problem (whose decidability status is unknown).

我们研究了带有数据的 Petri 网，Petri 网的扩展，其中令牌携带来自无限数据域的值，并且过渡的可执行性取决于数据值之间的等式。我们提供了一个双向可达性问题的决策过程：给定一个 Petri 网及其两个配置，我们询问每个配置是否可以从另一个配置达到。这推动了可判定性边界，因为双向可达性问题包含可覆盖性问题（已知可判定）并被可达性问题所包含（其可判定性状态未知）。

Petri 网中的双可达性

Bi-reachability in Petri nets with data

The generalization problem on KBQA has drawn considerable attention. Existing
research suffers from the generalization issue brought by the entanglement in
the coarse-grained modeling of the logical expression, or inexecutability
issues due to the fine-grained modeling of disconnected classes and relations
in real KBs. We propose a Fine-to-Coarse Composition framework for KBQA
(FC-KBQA) to both ensure the generalization ability and executability of the
logical expression. The main idea of FC-KBQA is to extract relevant
fine-grained knowledge components from KB and reformulate them into
middle-grained knowledge pairs for generating the final logical expressions.
FC-KBQA derives new state-of-the-art performance on GrailQA and WebQSP, and
runs 4 times faster than the baseline.

本文提出了一种基于精细到粗粒度组合的 KBQA 框架 (FC-KBQA)，它通过从 KB 中提取相关的细粒度的知识组件并将其重新构成中粒度的知识对，以实现逻辑表达式的泛化能力和可执行性。实验证明，FC-KBQA 在 GrailQA 和 WebQSP 上都有最先进的性能表现，并且比基线系统运行速度快 4 倍。