The problem of natural language generation, and, more specifically, method
name prediction, faces significant difficulties when proposed models need to be
evaluated on test data. Such a metric would need to consider the versatility
with which a single method can be named, with respect to both semantics and
syntax. Measuring the direct overlap between the predicted and reference (true)
sequences will not be able to capture these subtleties. Other existing
embedding based metrics either do not measure precision and recall or impose
strict unrealistic assumptions on both sequences. To address these issues, we
propose a new metric that, on the one hand, is very simple and lightweight,
and, on the other hand, is able to calculate precision and recall without
resorting to any assumptions while obtaining good performance with respect to
the human judgement.

自然语言生成中方法名称预测面临困难，为了解决这些问题，我们提出了一种新的度量标准，能够计算精确度和召回率，并在与人类判断相比获得良好的性能。

WRDScore: 评估自然语言生成模型的新指标

WRDScore: New Metric for Evaluation of Natural Language Generation  Models

Learning from source code usually requires a large amount of labeled data.
Despite the possible scarcity of labeled data, the trained model is highly
task-specific and lacks transferability to different tasks. In this work, we
present effective pre-training strategies on top of a novel graph-based code
representation, to produce universal representations for code. Specifically,
our graph-based representation captures important semantics between code
elements (e.g., control flow and data flow). We pre-train graph neural networks
on the representation to extract universal code properties. The pre-trained
model then enables the possibility of fine-tuning to support various downstream
applications. We evaluate our model on two real-world datasets -- spanning over
30M Java methods and 770K Python methods. Through visualization, we reveal
discriminative properties in our universal code representation. By comparing
multiple benchmarks, we demonstrate that the proposed framework achieves
state-of-the-art results on method name prediction and code graph link
prediction.

本研究提出了一种基于图形的代码表示技术，并通过有效的预训练策略生成通用代码表示。通过在真实数据集上的测试，表明该方法在方法名称预测和代码图链接预测方面具有最先进的结果。

代码的通用表示方法

Universal Representation for Code

With the prevalence of publicly available source code repositories to train
deep neural network models, neural program models can do well in source code
analysis tasks such as predicting method names in given programs that cannot be
easily done by traditional program analysis techniques. Although such neural
program models have been tested on various existing datasets, the extent to
which they generalize to unforeseen source code is largely unknown. Since it is
very challenging to test neural program models on all unforeseen programs, in
this paper, we propose to evaluate the generalizability of neural program
models with respect to semantic-preserving transformations: a generalizable
neural program model should perform equally well on programs that are of the
same semantics but of different lexical appearances and syntactical structures.
We compare the results of various neural program models for the method name
prediction task on programs before and after automated semantic-preserving
transformations. We use three Java datasets of different sizes and three
state-of-the-art neural network models for code, namely code2vec, code2seq, and
GGNN, to build nine such neural program models for evaluation. Our results show
that even with small semantically preserving changes to the programs, these
neural program models often fail to generalize their performance. Our results
also suggest that neural program models based on data and control dependencies
in programs generalize better than neural program models based only on abstract
syntax trees. On the positive side, we observe that as the size of the training
dataset grows and diversifies the generalizability of correct predictions
produced by the neural program models can be improved too. Our results on the
generalizability of neural program models provide insights to measure their
limitations and provide a stepping stone for their improvement.

本文通过对三个 Java 数据集和三种最先进的神经网络代码的比较，来评估神经程序模型在语义保持转换下的一般性，并发现神经程序模型在语法和外部形象不同但语义相同的程序上表现不佳。