Prediction head is a crucial component of Transformer language models.
Despite its direct impact on prediction, this component has often been
overlooked in analyzing Transformers. In this study, we investigate the inner
workings of the prediction head, specifically focusing on bias parameters. Our
experiments with BERT and GPT-2 models reveal that the biases in their word
prediction heads play a significant role in the models' ability to reflect word
frequency in a corpus, aligning with the logit adjustment method commonly used
in long-tailed learning. We also quantify the effect of controlling the biases
in practical auto-regressive text generation scenarios; under a particular
setting, more diverse text can be generated without compromising text quality.

本研究调查了预测头的内部工作，特别关注偏差参数，并发现 BERT 和 GPT-2 模型中的预测头偏差对模型反映语料库中的词频有重要作用，可通过对偏差的控制在实践自回归文本生成场景下生成更多样化的文本。

Transformer 语言模型处理预测头中的词频

Transformer Language Models Handle Word Frequency in Prediction Head

Learning to predict properties of large graphs is challenging because each
prediction requires the knowledge of an entire graph, while the amount of
memory available during training is bounded. Here we propose Graph Segment
Training (GST), a general framework that utilizes a divide-and-conquer approach
to allow learning large graph property prediction with a constant memory
footprint. GST first divides a large graph into segments and then
backpropagates through only a few segments sampled per training iteration. We
refine the GST paradigm by introducing a historical embedding table to
efficiently obtain embeddings for segments not sampled for backpropagation. To
mitigate the staleness of historical embeddings, we design two novel
techniques. First, we finetune the prediction head to fix the input
distribution shift. Second, we introduce Stale Embedding Dropout to drop some
stale embeddings during training to reduce bias. We evaluate our complete
method GST-EFD (with all the techniques together) on two large graph property
prediction benchmarks: MalNet and TpuGraphs. Our experiments show that GST-EFD
is both memory-efficient and fast, while offering a slight boost on test
accuracy over a typical full graph training regime.

该研究提出了一种名为 Graph Segment Training (GST) 的框架，它通过历史嵌入表获取大型图形的嵌入，实现了基于分段策略的预测学习方法，从而提高了效率和准确性。

通过图段训练学习大型图的属性预测

Learning Large Graph Property Prediction via Graph Segment Training

Recently the surprising discovery of the Bootstrap Your Own Latent (BYOL)
method by Grill et al. shows the negative term in contrastive loss can be
removed if we add the so-called prediction head to the network. This initiated
the research of non-contrastive self-supervised learning. It is mysterious why
even when there exist trivial collapsed global optimal solutions, neural
networks trained by (stochastic) gradient descent can still learn competitive
representations. This phenomenon is a typical example of implicit bias in deep
learning and remains little understood.
In this work, we present our empirical and theoretical discoveries on
non-contrastive self-supervised learning. Empirically, we find that when the
prediction head is initialized as an identity matrix with only its off-diagonal
entries being trainable, the network can learn competitive representations even
though the trivial optima still exist in the training objective. Theoretically,
we present a framework to understand the behavior of the trainable, but
identity-initialized prediction head. Under a simple setting, we characterized
the substitution effect and acceleration effect of the prediction head. The
substitution effect happens when learning the stronger features in some neurons
can substitute for learning these features in other neurons through updating
the prediction head. And the acceleration effect happens when the substituted
features can accelerate the learning of other weaker features to prevent them
from being ignored. These two effects enable the neural networks to learn all
the features rather than focus only on learning the stronger features, which is
likely the cause of the dimensional collapse phenomenon. To the best of our
knowledge, this is also the first end-to-end optimization guarantee for
non-contrastive methods using nonlinear neural networks with a trainable
prediction head and normalization.

本文介绍了一种自监督学习方法，其中相称损失中的负项可以通过添加 prediction head 被消除，理论上和实验上证明了这种方法能够避免维度崩溃现象，并且通过正常的梯度下降可以学习有效表示来替代只学习强特征的偏见。

无对比自监督学习中预测头机制

The Mechanism of Prediction Head in Non-contrastive Self-supervised  Learning

The explosive growth of image data facilitates the fast development of image
processing and computer vision methods for emerging visual applications,
meanwhile introducing novel distortions to the processed images. This poses a
grand challenge to existing blind image quality assessment (BIQA) models,
failing to continually adapt to such subpopulation shift. Recent work suggests
training BIQA methods on the combination of all available human-rated IQA
datasets. However, this type of approach is not scalable to a large number of
datasets, and is cumbersome to incorporate a newly created dataset as well. In
this paper, we formulate continual learning for BIQA, where a model learns
continually from a stream of IQA datasets, building on what was learned from
previously seen data. We first identify five desiderata in the new setting with
a measure to quantify the plasticity-stability trade-off. We then propose a
simple yet effective method for learning BIQA models continually. Specifically,
based on a shared backbone network, we add a prediction head for a new dataset,
and enforce a regularizer to allow all prediction heads to evolve with new data
while being resistant to catastrophic forgetting of old data. We compute the
quality score by an adaptive weighted summation of estimates from all
prediction heads. Extensive experiments demonstrate the promise of the proposed
continual learning method in comparison to standard training techniques for
BIQA. We made the code publicly available at
this https URL

本研究提出了一种针对 BLQA 的基于连续学习的模型，以提高模型的适应性和稳定性，通过新的预测头和正则化器来实现在不断增长的数据流中不断学习并避免灾难性遗忘，在所有预测头的估计的自适应加权求和得出最终图像质量评分。