To support software developers in understanding and maintaining programs,
various automatic code summarization techniques have been proposed to generate
a concise natural language comment for a given code snippet. Recently, the
emergence of large language models (LLMs) has led to a great boost in the
performance of natural language processing tasks. Among them, ChatGPT is the
most popular one which has attracted wide attention from the software
engineering community. However, it still remains unclear how ChatGPT performs
in (automatic) code summarization. Therefore, in this paper, we focus on
evaluating ChatGPT on a widely-used Python dataset called CSN-Python and
comparing it with several state-of-the-art (SOTA) code summarization models.
Specifically, we first explore an appropriate prompt to guide ChatGPT to
generate in-distribution comments. Then, we use such a prompt to ask ChatGPT to
generate comments for all code snippets in the CSN-Python test set. We adopt
three widely-used metrics (including BLEU, METEOR, and ROUGE-L) to measure the
quality of the comments generated by ChatGPT and SOTA models (including NCS,
CodeBERT, and CodeT5). The experimental results show that in terms of BLEU and
ROUGE-L, ChatGPT's code summarization performance is significantly worse than
all three SOTA models. We also present some cases and discuss the advantages
and disadvantages of ChatGPT in code summarization. Based on the findings, we
outline several open challenges and opportunities in ChatGPT-based code
summarization.

本文重点评估了最流行的大型语言模型 ChatGPT 在自动代码注释中的性能，并将其与其他最先进的代码摘要模型进行比较，结果表明在 BLEU 和 ROUGE-L 方面，ChatGPT 的代码摘要性能显着低于其他三个模型。

通过 ChatGPT 的自动代码摘要：我们还有多远？

Automatic Code Summarization via ChatGPT: How Far Are We?

Code execution is a fundamental aspect of programming language semantics that
reflects the exact behavior of the code. However, most pre-trained models for
code intelligence ignore the execution trace and only rely on source code and
syntactic structures. In this paper, we investigate how well pre-trained models
can understand and perform code execution. We develop a mutation-based data
augmentation technique to create a large-scale and realistic Python dataset and
task for code execution, which challenges existing models such as Codex. We
then present CodeExecutor, a Transformer model that leverages code execution
pre-training and curriculum learning to enhance its semantic comprehension. We
evaluate CodeExecutor on code execution and show its promising performance and
limitations. We also demonstrate its potential benefits for code intelligence
tasks such as zero-shot code-to-code search and text-to-code generation. Our
analysis provides insights into the learning and generalization abilities of
pre-trained models for code execution.

本文介绍了一个使用变异数据增强技术创建 Python 数据集并基于 Transformer 的 CodeExecutor 模型来强化其语义理解的方法，该模型主要用于代码执行任务。作者评估了 CodeExecutor 的性能和局限，并展示了它在零样本代码到代码搜索和文本到代码生成等方面的潜在好处。

使用预训练语言模型进行代码执行

Code Execution with Pre-trained Language Models

We advance the state-of-the-art in the accuracy of code prediction (next
token prediction) used in autocomplete systems. First, we report that using the
recently proposed Transformer architecture even out-of-the-box outperforms
previous neural and non-neural systems for code prediction. We then show that
by making the Transformer architecture aware of the syntactic structure of
code, we further increase the margin by which a Transformer-based system
outperforms previous systems. With this, it outperforms the accuracy of an
RNN-based system (similar to Hellendoorn et al. 2018) by 18.3%, the Deep3
system (Raychev et al 2016) by 14.1%, and an adaptation of Code2Seq (Alon et
al., 2018) for code prediction by 14.4%.
We present in the paper several ways of communicating the code structure to
the Transformer, which is fundamentally built for processing sequence data. We
provide a comprehensive experimental evaluation of our proposal, along with
alternative design choices, on a standard Python dataset, as well as on a
Facebook internal Python corpus. Our code and data preparation pipeline will be
available in open source.

该研究使用 Transformer 架构，通过让其了解代码的句法结构，提高了预测代码的准确性，在标准的 Python 数据集和 Facebook 内部 Python 语料库上进行了全面的实验评估。