Pretrained Transformers (PT) have been shown to improve Out of Distribution
(OOD) robustness than traditional models such as Bag of Words (BOW), LSTMs,
Convolutional Neural Networks (CNN) powered by Word2Vec and Glove embeddings.
How does the robustness comparison hold in a real world setting where some part
of the dataset can be noisy? Do PT also provide more robust representation than
traditional models on exposure to noisy data? We perform a comparative study on
10 models and find an empirical evidence that PT provide less robust
representation than traditional models on exposure to noisy data. We
investigate further and augment PT with an adversarial filtering (AF) mechanism
that has been shown to improve OOD generalization. However, increase in
generalization does not necessarily increase robustness, as we find that noisy
data fools the AF method powered by PT.

对 10 种模型进行对比研究，发现在加入噪声数据的情况下，预训练 Transformer 模型比传统模型提供了更少的鲁棒性，在使用对抗过滤方法进行扩展后，虽然增加了泛化性，但在面对噪声数据时依然无法提供足够的鲁棒性。

预训练的转换器并不总是能提高健壮性

Pretrained Transformers Do not Always Improve Robustness

Recently, test time adaptation (TTA) has attracted increasing attention due
to its power of handling the distribution shift issue in the real world. Unlike
what has been developed for convolutional neural networks (CNNs) for image
data, TTA is less explored for Graph Neural Networks (GNNs). There is still a
lack of efficient algorithms tailored for graphs with irregular structures. In
this paper, we present a novel test time adaptation strategy named Graph
Adversarial Pseudo Group Contrast (GAPGC), for graph neural networks TTA, to
better adapt to the Out Of Distribution (OOD) test data. Specifically, GAPGC
employs a contrastive learning variant as a self-supervised task during TTA,
equipped with Adversarial Learnable Augmenter and Group Pseudo-Positive Samples
to enhance the relevance between the self-supervised task and the main task,
boosting the performance of the main task. Furthermore, we provide theoretical
evidence that GAPGC can extract minimal sufficient information for the main
task from information theory perspective. Extensive experiments on molecular
scaffold OOD dataset demonstrated that the proposed approach achieves
state-of-the-art performance on GNNs.

本文提出了一种名为 GAPGC 的新型测试时间适应策略，通过对比学习和自监督学习等方法，有效增强图神经网络的适应性和性能，并且提供了信息论的理论证明。实验结果表明，该方法在分子骨架 OOD 数据集上取得了最先进的 GNN 性能。

GraphTTA: 图神经网络测试时间自适应

GraphTTA: Test Time Adaptation on Graph Neural Networks

Curriculum Learning is a powerful training method that allows for faster and
better training in some settings. This method, however, requires having a
notion of which examples are difficult and which are easy, which is not always
trivial to provide. A recent metric called C-Score acts as a proxy for example
difficulty by relating it to learning consistency. Unfortunately, this method
is quite compute intensive which limits its applicability for alternative
datasets. In this work, we train models through different methods to predict
C-Score for CIFAR-100 and CIFAR-10. We find, however, that these models
generalize poorly both within the same distribution as well as out of
distribution. This suggests that C-Score is not defined by the individual
characteristics of each sample but rather by other factors. We hypothesize that
a sample's relation to its neighbours, in particular, how many of them share
the same labels, can help in explaining C-Scores. We plan to explore this in
future work.

本文研究如何通过训练模型来预测 CIFAR-100 和 CIFAR-10 的难度得分，发现 C-Score 难以泛化到其他数据集，因此提出基于样本关系的假设，希望在进一步的工作中探索。