Large language models (LLMs) such as Llama 2 perform very well on tasks that
involve both natural language and source code, particularly code summarization
and code generation. We show that for the task of code summarization, the
performance of these models on individual examples often depends on the amount
of (subword) token overlap between the code and the corresponding reference
natural language descriptions in the dataset. This token overlap arises because
the reference descriptions in standard datasets (corresponding to docstrings in
large code bases) are often highly similar to the names of the functions they
describe. We also show that this token overlap occurs largely in the function
names of the code and compare the relative performance of these models after
removing function names versus removing code structure. We also show that using
multiple evaluation metrics like BLEU and BERTScore gives us very little
additional insight since these metrics are highly correlated with each other.

大语言模型在代码摘要任务方面，特别是代码生成和摘要具有很高的性能。本文发现，这些模型在每个示例上的性能往往取决于代码和对应参考自然语言描述之间的（子词）标记重叠量。此标记重叠主要出现在代码的函数名称中，并通过移除函数名称与移除代码结构来比较这些模型的相对性能。另外，使用 BLEU 和 BERTScore 等多个评估指标对此问题的洞见非常有限，因为这些指标高度相关。

大型语言模型在代码摘要上的性能分析

Analyzing the Performance of Large Language Models on Code Summarization

Pre-trained multilingual language models such as mBERT and XLM-R have
demonstrated great potential for zero-shot cross-lingual transfer to low
web-resource languages (LRL). However, due to limited model capacity, the large
difference in the sizes of available monolingual corpora between high
web-resource languages (HRL) and LRLs does not provide enough scope of
co-embedding the LRL with the HRL, thereby affecting downstream task
performance of LRLs. In this paper, we argue that relatedness among languages
in a language family along the dimension of lexical overlap may be leveraged to
overcome some of the corpora limitations of LRLs. We propose Overlap BPE
(OBPE), a simple yet effective modification to the BPE vocabulary generation
algorithm which enhances overlap across related languages. Through extensive
experiments on multiple NLP tasks and datasets, we observe that OBPE generates
a vocabulary that increases the representation of LRLs via tokens shared with
HRLs. This results in improved zero-shot transfer from related HRLs to LRLs
without reducing HRL representation and accuracy. Unlike previous studies that
dismissed the importance of token-overlap, we show that in the low-resource
related language setting, token overlap matters. Synthetically reducing the
overlap to zero can cause as much as a four-fold drop in zero-shot transfer
accuracy.

本文提出了一种名为 Overlap BPE 的用于增强低网络资源语言在多语言模型中表示的词汇生成算法，并通过多项实验表明，通过增加相关语言之间的词汇重叠度，可以在不降低高网络资源语言表示和准确性的前提下，提高相关低网络资源语言的零样本学习的效果。