In the realm of deep learning, understanding the latent space of language
models (LMs) like transformers is crucial for refining their performance and
interpretability. However, existing analyses often fall short in providing
absolute and model-centric insights into LM semantics, and neglect essential
aspects of LM adaption. In response, we introduce a pioneering method called
vocabulary-defined semantics, which establishes a fixed reference frame within
the LM latent space, ensuring absolute semantic analysis grounded in LM
vocabulary. Our approach transcends prior relative analyses, leveraging LM
vocabulary for model-centric insights. Furthermore, we propose a novel
technique to compute logits, emphasizing differentiability and local isotropy,
and introduce a neural clustering module for semantically calibrating data
representations during LM adaptation. Through extensive experiments across
diverse text understanding datasets, our approach surpasses state-of-the-art
methods of retrieval-augmented generation and parameters-efficient finetuning,
showcasing its efficacy and broad applicability. Our findings not only shed
light on LM mechanics but also offer practical solutions for enhancing LM
performance and interpretability.

我们介绍了一种名为 “基于词汇定义的语义” 的开创性方法，它在语言模型（LMs）的潜空间中建立了一个固定的参考框架，确保了在 LM 词汇基础上的绝对语义分析。通过广泛的实验，我们的方法超越了检索增强生成和参数有效微调的最先进方法，展示了它的功效和广泛适用性。

关于语言模型潜空间的语义：一种基于词汇定义的方法

On the Semantics of LM Latent Space: A Vocabulary-defined Approach

We present CLUSTSEG, a general, transformer-based framework that tackles
different image segmentation tasks (i.e., superpixel, semantic, instance, and
panoptic) through a unified neural clustering scheme. Regarding queries as
cluster centers, CLUSTSEG is innovative in two aspects:1) cluster centers are
initialized in heterogeneous ways so as to pointedly address task-specific
demands (e.g., instance- or category-level distinctiveness), yet without
modifying the architecture; and 2) pixel-cluster assignment, formalized in a
cross-attention fashion, is alternated with cluster center update, yet without
learning additional parameters. These innovations closely link CLUSTSEG to EM
clustering and make it a transparent and powerful framework that yields
superior results across the above segmentation tasks.

CLUSTSEG 是一个基于 transformer 的通用框架，通过统一的神经聚类方案解决不同的图像分割任务，并通过两个创新方面实现。像 EM 聚类一样，这些创新使 CLUSTSEG 成为一个透明而强大的框架，在上述分割任务中产生优越的结果。