We investigate compositional structures in vector data embeddings from
pre-trained vision-language models (VLMs). Traditionally, compositionality has
been associated with algebraic operations on embeddings of words from a
pre-existing vocabulary. In contrast, we seek to approximate label
representations from a text encoder as combinations of a smaller set of vectors
in the embedding space. These vectors can be seen as "ideal words" which can be
used to generate new concepts in an efficient way. We present a theoretical
framework for understanding linear compositionality, drawing connections with
mathematical representation theory and previous definitions of disentanglement.
We provide theoretical and empirical evidence that ideal words provide good
compositional approximations of composite concepts and can be more effective
than token-based decompositions of the same concepts.

本文讨论在预先训练的视觉与语言模型中向量数据嵌入的组合结构，提出一种基于理想词向量的新的合成性表达方法，并探讨其与数学表示理论和去耦合表示概念的联系。作者提供理论和经验证据表明，理想词嵌入提供良好的组合近似和比基于标记的分解更有效的组合表示方法。