The representation degeneration problem is a phenomenon that is widely
observed among self-supervised learning methods based on Transformers. In NLP,
it takes the form of anisotropy, a singular property of hidden representations
which makes them unexpectedly close to each other in terms of angular distance
(cosine-similarity). Some recent works tend to show that anisotropy is a
consequence of optimizing the cross-entropy loss on long-tailed distributions
of tokens. We show in this paper that anisotropy can also be observed
empirically in language models with specific objectives that should not suffer
directly from the same consequences. We also show that the anisotropy problem
extends to Transformers trained on other modalities. Our observations suggest
that anisotropy is actually inherent to Transformers-based models.

通过实证观察，本文展示了基于 Transformer 的语言模型和其他模态下的 Transformer 存在的角度距离接近的问题，即各向异性问题。

自注意力中的各向异性

Anisotropy Is Inherent to Self-Attention in Transformers

The representation degeneration problem is a phenomenon that is widely
observed among self-supervised learning methods based on Transformers. In NLP,
it takes the form of anisotropy, a singular property of hidden representations
which makes them unexpectedly close to each other in terms of angular distance
(cosine-similarity). Some recent works tend to show that anisotropy is a
consequence of optimizing the cross-entropy loss on long-tailed distributions
of tokens. We show in this paper that anisotropy can also be observed
empirically in language models with specific objectives that should not suffer
directly from the same consequences. We also show that the anisotropy problem
extends to Transformers trained on other modalities. Our observations tend to
demonstrate that anisotropy might actually be inherent to Transformers-based
models.

本文探讨了 self-supervised learning 方法在 transformer 架构的模型中出现的表示退化问题，提到了表征中的各向异性现象，并且指出该问题在表示学习、自然语言处理等领域都具有普适性。

Transformer 模型中的各向异性是否固有？

Is Anisotropy Inherent to Transformers?

Despite advances in multilingual neural machine translation (MNMT), we argue
that there are still two major challenges in this area: data imbalance and
representation degeneration. The data imbalance problem refers to the imbalance
in the amount of parallel corpora for all language pairs, especially for
long-tail languages (i.e., very low-resource languages). The representation
degeneration problem refers to the problem of encoded tokens tending to appear
only in a small subspace of the full space available to the MNMT model. To
solve these two issues, we propose Bi-ACL, a framework that uses only
target-side monolingual data and a bilingual dictionary to improve the
performance of the MNMT model. We define two modules, named bidirectional
autoencoder and bidirectional contrastive learning, which we combine with an
online constrained beam search and a curriculum learning sampling strategy.
Extensive experiments show that our proposed method is more effective both in
long-tail languages and in high-resource languages. We also demonstrate that
our approach is capable of transferring knowledge between domains and languages
in zero-shot scenarios.

提出了一个名为 Bi-ACL 的框架，旨在解决多语言神经机器翻译中的数据不平衡和表示退化问题。该框架使用仅目标侧单语数据和双语词典，结合双向自编码器和双向对比学习模块，并使用在线约束波束搜索和课程学习采样策略。实验表明，该方法对长尾语言和高资源语言都更有效，并且可以在零 - shot 场景中在不同领域和语言之间转移知识。