Autoregressive language models have demonstrated a remarkable ability to
extract latent structure from text. The embeddings from large language models
have been shown to capture aspects of the syntax and semantics of language. But
what {\em should} embeddings represent? We connect the autoregressive
prediction objective to the idea of constructing predictive sufficient
statistics to summarize the information contained in a sequence of
observations, and use this connection to identify three settings where the
optimal content of embeddings can be identified: independent identically
distributed data, where the embedding should capture the sufficient statistics
of the data; latent state models, where the embedding should encode the
posterior distribution over states given the data; and discrete hypothesis
spaces, where the embedding should reflect the posterior distribution over
hypotheses given the data. We then conduct empirical probing studies to show
that transformers encode these three kinds of latent generating distributions,
and that they perform well in out-of-distribution cases and without token
memorization in these settings.

通过将自回归预测目标与构建预测充分统计量的思想联系起来，我们确定了三种情况下嵌入的最佳内容：独立同分布数据、潜在状态模型和离散假设空间，并进行实证研究表明 Transformers 编码了这三种潜在生成分布，并在这些情况下表现良好。