The rapid scaling of large language models (LLMs) has raised concerns about
the transparency and fair use of the pretraining data used for training them.
Detecting such content is challenging due to the scale of the data and limited
exposure of each instance during training. We propose ReCaLL (Relative
Conditional Log-Likelihood), a novel membership inference attack (MIA) to
detect LLMs' pretraining data by leveraging their conditional language modeling
capabilities. ReCaLL examines the relative change in conditional
log-likelihoods when prefixing target data points with non-member context. Our
empirical findings show that conditioning member data on non-member prefixes
induces a larger decrease in log-likelihood compared to non-member data. We
conduct comprehensive experiments and show that ReCaLL achieves
state-of-the-art performance on the WikiMIA dataset, even with random and
synthetic prefixes, and can be further improved using an ensemble approach.
Moreover, we conduct an in-depth analysis of LLMs' behavior with different
membership contexts, providing insights into how LLMs leverage membership
information for effective inference at both the sequence and token level.

通过条件语言建模开展一个新的成员推理攻击，称为 ReCaLL，以检测大型语言模型的预训练数据，并展示其在 WikiMIA 数据集上能够达到最先进的性能。

ReCaLL：基于相对条件对数似然的成员推断

ReCaLL: Membership Inference via Relative Conditional Log-Likelihoods

Generative Adversarial Networks (GANs) have gathered a lot of attention from
the computer vision community, yielding impressive results for image
generation. Advances in the adversarial generation of natural language from
noise however are not commensurate with the progress made in generating images,
and still lag far behind likelihood based methods. In this paper, we take a
step towards generating natural language with a GAN objective alone. We
introduce a simple baseline that addresses the discrete output space problem
without relying on gradient estimators and show that it is able to achieve
state-of-the-art results on a Chinese poem generation dataset. We present
quantitative results on generating sentences from context-free and
probabilistic context-free grammars, and qualitative language modeling results.
A conditional version is also described that can generate sequences conditioned
on sentence characteristics.

本文探讨了使用 GAN 自学习等机制解决自然语言生成中离散输出空间问题；通过实验表明，提出的模型在中文古诗生成数据集上取得了最新的高分，可以生成自由语序和概率性上下文无关语法。此外还介绍了一个条件模型，可根据句子特征生成序列。