Large language models are trained on vast amounts of internet data, prompting
concerns and speculation that they have memorized public benchmarks. Going from
speculation to proof of contamination is challenging, as the pretraining data
used by proprietary models are often not publicly accessible. We show that it
is possible to provide provable guarantees of test set contamination in
language models without access to pretraining data or model weights. Our
approach leverages the fact that when there is no data contamination, all
orderings of an exchangeable benchmark should be equally likely. In contrast,
the tendency for language models to memorize example order means that a
contaminated language model will find certain canonical orderings to be much
more likely than others. Our test flags potential contamination whenever the
likelihood of a canonically ordered benchmark dataset is significantly higher
than the likelihood after shuffling the examples. We demonstrate that our
procedure is sensitive enough to reliably prove test set contamination in
challenging situations, including models as small as 1.4 billion parameters, on
small test sets of only 1000 examples, and datasets that appear only a few
times in the pretraining corpus. Using our test, we audit five popular publicly
accessible language models for test set contamination and find little evidence
for pervasive contamination.

通过无需预训练数据或模型权重的方法，我们可以提供对语言模型测试集污染的可证明保证，通过对典型排序的基准数据集的似然性进行比较，我们的测试能够可靠地证明测试集污染的情况。在五个常见的公开可访问的语言模型中，我们的测试发现很少有普遍污染的证据。

在黑盒语言模型中证明测试集污染

Proving Test Set Contamination in Black Box Language Models

We propose a new family of efficient and expressive deep generative models of
graphs, called Graph Recurrent Attention Networks (GRANs). Our model generates
graphs one block of nodes and associated edges at a time. The block size and
sampling stride allow us to trade off sample quality for efficiency. Compared
to previous RNN-based graph generative models, our framework better captures
the auto-regressive conditioning between the already-generated and
to-be-generated parts of the graph using Graph Neural Networks (GNNs) with
attention. This not only reduces the dependency on node ordering but also
bypasses the long-term bottleneck caused by the sequential nature of RNNs.
Moreover, we parameterize the output distribution per block using a mixture of
Bernoulli, which captures the correlations among generated edges within the
block. Finally, we propose to handle node orderings in generation by
marginalizing over a family of canonical orderings. On standard benchmarks, we
achieve state-of-the-art time efficiency and sample quality compared to
previous models. Additionally, we show our model is capable of generating large
graphs of up to 5K nodes with good quality. To the best of our knowledge, GRAN
is the first deep graph generative model that can scale to this size. Our code
is released at: this https URL

本文提出了一种名为 GRAN 的新型图形回归注意力网络的有效且表达性强的深度生成模型，能够生成高质量大规模图形，同时与之前的 RNN-based 生成模型相比，GRAN 使用了 GNN 和注意力机制更好地捕捉到已生成和待生成部分的条件之间的自回归关系，并在输出分布上采用伯努利分布的混合来捕捉区块内的生成边之间的相关性，同时在处理节点顺序方面也提出了一些新的解决方法。

基于图循环注意力网络的高效图生成

Efficient Graph Generation with Graph Recurrent Attention Networks

We consider a simple and overarching representation for permutation-invariant
functions of sequences (or multiset functions). Our approach, which we call
Janossy pooling, expresses a permutation-invariant function as the average of a
permutation-sensitive function applied to all reorderings of the input
sequence. This allows us to leverage the rich and mature literature on
permutation-sensitive functions to construct novel and flexible
permutation-invariant functions. If carried out naively, Janossy pooling can be
computationally prohibitive. To allow computational tractability, we consider
three kinds of approximations: canonical orderings of sequences, functions with
$k$-order interactions, and stochastic optimization algorithms with random
permutations. Our framework unifies a variety of existing work in the
literature, and suggests possible modeling and algorithmic extensions. We
explore a few in our experiments, which demonstrate improved performance over
current state-of-the-art methods.

本文提出了 Janossy pooling 方法，可以将置换不变函数表示为输入序列的所有重新排序应用置换敏感函数的平均值，并且通过三种近似方法实现了可计算性可控，实验结果表明该方法在置换不变函数的构建上表现出了比当前最先进的方法更好的性能。