We propose to extract meaning representations from autoregressive language
models by considering the distribution of all possible trajectories extending
an input text. This strategy is prompt-free, does not require fine-tuning, and
is applicable to any pre-trained autoregressive model. Moreover, unlike
vector-based representations, distribution-based representations can also model
asymmetric relations (e.g., direction of logical entailment, hypernym/hyponym
relations) by using algebraic operations between likelihood functions. These
ideas are grounded in distributional perspectives on semantics and are
connected to standard constructions in automata theory, but to our knowledge
they have not been applied to modern language models. We empirically show that
the representations obtained from large models align well with human
annotations, outperform other zero-shot and prompt-free methods on semantic
similarity tasks, and can be used to solve more complex entailment and
containment tasks that standard embeddings cannot handle. Finally, we extend
our method to represent data from different modalities (e.g., image and text)
using multimodal autoregressive models.

我们提出了一种从自回归语言模型中提取含义表示的方法，通过考虑扩展输入文本的所有可能轨迹的分布来实现。这种策略是无提示的，不需要微调，并且适用于任何预训练的自回归模型。此外，与基于向量的表示不同，基于分布的表示还可以通过使用似然函数之间的代数运算来建模非对称关系（例如，逻辑蕴含的方向，上位词 / 下位词关系）。这些思想扎根于语义上的分布观点，并与自动机理论中的标准构造相连接，但据我们所知，它们尚未应用于现代语言模型。我们通过实验证明，从大型模型获得的表示与人类注释相匹配，比其他零样本和无提示方法在语义相似性任务上表现更好，并且可以用于解决标准嵌入不能处理的更复杂的蕴含和包含任务。最后，我们将我们的方法扩展到使用多模态自回归模型表示来自不同模态（例如，图像和文本）的数据。