We find limits to the Transformer architecture for language modeling and show
it has a universal prediction property in an information-theoretic sense. We
further analyze performance in non-asymptotic data regimes to understand the
role of various components of the Transformer architecture, especially in the
context of data-efficient training. We validate our theoretical analysis with
experiments on both synthetic and real datasets.

本研究发现了 Transformer 架构在语言模型方面的局限性，证明了它在信息理论意义上具有普适预测性，并在非渐近数据区域中分析了各种 Transformer 架构组件的性能，尤其是在数据有效训练的情境中。我们通过对合成和真实数据集的实验验证了我们的理论分析。

变压器是通用预测器

Transformers are Universal Predictors

Recent advances in semi-supervised learning (SSL) demonstrate that a
combination of consistency regularization and pseudo-labeling can effectively
improve image classification accuracy in the low-data regime. Compared to
classification, semantic segmentation tasks require much more intensive
labeling costs. Thus, these tasks greatly benefit from data-efficient training
methods. However, structured outputs in segmentation render particular
difficulties (e.g., designing pseudo-labeling and augmentation) to apply
existing SSL strategies. To address this problem, we present a simple and novel
re-design of pseudo-labeling to generate well-calibrated structured pseudo
labels for training with unlabeled or weakly-labeled data. Our proposed
pseudo-labeling strategy is network structure agnostic to apply in a one-stage
consistency training framework. We demonstrate the effectiveness of the
proposed pseudo-labeling strategy in both low-data and high-data regimes.
Extensive experiments have validated that pseudo labels generated from wisely
fusing diverse sources and strong data augmentation are crucial to consistency
training for segmentation. The source code is available at
this https URL

本文提出了一种利用伪标签进行半监督语义分割训练的策略，通过合理融合不同来源和强数据增强来生成实验结果验证有效性。