Deep Neural Networks (DNNs) are prone to learn shortcut patterns that damage
the generalization of the DNN during deployment. Shortcut Learning is
concerning, particularly when the DNNs are applied to safety-critical domains.
This paper aims to better understand shortcut learning through the lens of the
learning dynamics of the internal neurons during the training process. More
specifically, we make the following observations: (1) While previous works
treat shortcuts as synonymous with spurious correlations, we emphasize that not
all spurious correlations are shortcuts. We show that shortcuts are only those
spurious features that are "easier" than the core features. (2) We build upon
this premise and use instance difficulty methods (like Prediction Depth) to
quantify "easy" and to identify this behavior during the training phase. (3) We
empirically show that shortcut learning can be detected by observing the
learning dynamics of the DNN's early layers, irrespective of the network
architecture used. In other words, easy features learned by the initial layers
of a DNN early during the training are potential shortcuts. We verify our
claims on simulated and real medical imaging data and justify the empirical
success of our hypothesis by showing the theoretical connections between
Prediction Depth and information-theoretic concepts like V-usable information.
Lastly, our experiments show the insufficiency of monitoring only accuracy
plots during training (as is common in machine learning pipelines), and we
highlight the need for monitoring early training dynamics using example
difficulty metrics.

本文通过观察 Deep Neural Networks (DNNs) 内部神经元的学习动态，提出了易学特征会导致 Shortcut Learning 的假设，并用实验验证了此假设。文章主张在早期的训练动态中监测模型的表现而非仅仅监测模型准确率。

早期训练动态视角下的快捷学习

Shortcut Learning Through the Lens of Early Training Dynamics

Early exiting allows instances to exit at different layers according to the
estimation of difficulty. Previous works usually adopt heuristic metrics such
as the entropy of internal outputs to measure instance difficulty, which
suffers from generalization and threshold-tuning. In contrast, learning to
exit, or learning to predict instance difficulty is a more appealing way.
Though some effort has been devoted to employing such "learn-to-exit" modules,
it is still unknown whether and how well the instance difficulty can be
learned. As a response, we first conduct experiments on the learnability of
instance difficulty, which demonstrates that modern neural models perform
poorly on predicting instance difficulty. Based on this observation, we propose
a simple-yet-effective Hash-based Early Exiting approach (HashEE) that replaces
the learn-to-exit modules with hash functions to assign each token to a fixed
exiting layer. Different from previous methods, HashEE requires no internal
classifiers nor extra parameters, and therefore is more efficient. Experimental
results on classification, regression, and generation tasks demonstrate that
HashEE can achieve higher performance with fewer FLOPs and inference time
compared with previous state-of-the-art early exiting methods.

本文提出了一种基于哈希技术的早期结束方法，即 HashEE 方法，以替代 learn-to-exit 模块来预测每个实例的困难度，并将每个令牌分配到一个固定的退出层。实验证明，与先前的早期退出方法相比，该方法在分类、回归和生成任务中能够实现更高的性能，且需要更少的 FLOPs 和推理时间，并且不需要内部分类器或额外参数。

一种基于简单哈希的语言理解和生成早期退出方法

A Simple Hash-Based Early Exiting Approach For Language Understanding and Generation

Modern NLP systems require high-quality annotated data. In specialized
domains, expert annotations may be prohibitively expensive. An alternative is
to rely on crowdsourcing to reduce costs at the risk of introducing noise. In
this paper we demonstrate that directly modeling instance difficulty can be
used to improve model performance, and to route instances to appropriate
annotators. Our difficulty prediction model combines two learned
representations: a `universal' encoder trained on out-of-domain data, and a
task-specific encoder. Experiments on a complex biomedical information
extraction task using expert and lay annotators show that: (i) simply excluding
from the training data instances predicted to be difficult yields a small boost
in performance; (ii) using difficulty scores to weight instances during
training provides further, consistent gains; (iii) assigning instances
predicted to be difficult to domain experts is an effective strategy for task
routing. Our experiments confirm the expectation that for specialized tasks
expert annotations are higher quality than crowd labels, and hence preferable
to obtain if practical. Moreover, augmenting small amounts of expert data with
a larger set of lay annotations leads to further improvements in model
performance.

本文研究如何通过模拟实例难度来提高专业领域 NLP 系统的表现，包括使用 crowdsourcing 降低成本，同时为优化任务路由分配特定领域的专家。试验结果显示，通过难度预测，可以在既有少量专家数据的基础上进一步优化模型性能。