Foundation models are trained on increasingly immense and opaque datasets.
Even while these models are now key in AI system building, it can be difficult
to answer the straightforward question: has the model already encountered a
given example during training? We therefore propose a widespread adoption of
Data Portraits: artifacts that record training data and allow for downstream
inspection. First we outline the properties of such an artifact and discuss how
existing solutions can be used to increase transparency. We then propose and
implement a solution based on data sketching, stressing fast and space
efficient querying. Using our tool, we document a popular large language
modeling corpus (the Pile) and show that our solution enables answering
questions about test set leakage and model plagiarism. Our tool is lightweight
and fast, costing only 3% of the dataset size in overhead. We release a demo of
our tools at this http URL and call on dataset and model creators to
release Data Portraits as a complement to current documentation practices.

提出了 “数据肖像” 的概念，以记录训练数据，以增加模型训练的透明度并解决模型抄袭、数据泄漏等问题。作者采用数据草图技术实现了该方案，仅增加 3% 的额外开销，在大规模语言模型数据集上证明了该方案的有效性。

数据肖像：记录基础模型训练数据

Data Portraits: Recording Foundation Model Training Data

Datasets that are terabytes in size are increasingly common, but computer
bottlenecks often frustrate a complete analysis of the data. While more data
are better than less, diminishing returns suggest that we may not need
terabytes of data to estimate a parameter or test a hypothesis. But which rows
of data should we analyze, and might an arbitrary subset of rows preserve the
features of the original data? This paper reviews a line of work that is
grounded in theoretical computer science and numerical linear algebra, and
which finds that an algorithmically desirable sketch, which is a randomly
chosen subset of the data, must preserve the eigenstructure of the data, a
property known as a subspace embedding. Building on this work, we study how
prediction and inference can be affected by data sketching within a linear
regression setup. We show that the sketching error is small compared to the
sample size effect which a researcher can control. As a sketch size that is
algorithmically optimal may not be suitable for prediction and inference, we
use statistical arguments to provide 'inference conscious' guides to the sketch
size. When appropriately implemented, an estimator that pools over different
sketches can be nearly as efficient as the infeasible one using the full
sample.

这篇论文研究数据分析中的线性回归和数据素描技术，探讨在样本量有限的情况下如何选择行列子集对数据进行估计和推断，发现虽然算法上的优化子集无法适用于预测和推断，但通过统计学方法可以提供建议的子集大小，并通过实验表明，使用不同的子集来估计结果可以将预测结果效率与使用全样本的方法几乎相同。