The learning dynamics of deep neural networks are not well understood. The
information bottleneck (IB) theory proclaimed separate fitting and compression
phases. But they have since been heavily debated. We comprehensively analyze
the learning dynamics by investigating a layer's reconstruction ability of the
input and prediction performance based on the evolution of parameters during
training. We empirically show the existence of three phases using common
datasets and architectures such as ResNet and VGG: (i) near constant
reconstruction loss, (ii) decrease, and (iii) increase. We also derive an
empirically grounded data model and prove the existence of phases for
single-layer networks. Technically, our approach leverages classical complexity
analysis. It differs from IB by relying on measuring reconstruction loss rather
than information theoretic measures to relate information of intermediate
layers and inputs. Our work implies a new best practice for transfer learning:
We show empirically that the pre-training of a classifier should stop well
before its performance is optimal.

通过对参数的演化，我们全面分析了深度神经网络的学习动态，发现存在三个阶段：接近恒定的重建损失、下降和上升。我们还通过经验实证建立了数据模型，并对单层神经网络证明了阶段的存在。我们的工作为迁移学习提供了新的最佳实践：通过实验证明预训练的分类器在性能达到最优之前应该停止。

理解和利用神经网络的学习阶段

Understanding and Leveraging the Learning Phases of Neural Networks

Transformer-based pretrained language models (PLMs) have achieved great
success in modern NLP. An important advantage of PLMs is good
out-of-distribution (OOD) robustness. Recently, diffusion models have attracted
a lot of work to apply diffusion to PLMs. It remains under-explored how
diffusion influences PLMs on OOD data. The core of diffusion models is a
forward diffusion process which gradually applies Gaussian noise to inputs, and
a reverse denoising process which removes noise. The noised input
reconstruction is a fundamental ability of diffusion models. We directly
analyze OOD robustness by measuring the reconstruction loss, including testing
the abilities to reconstruct OOD data, and to detect OOD samples. Experiments
are conducted by analyzing different training parameters and data statistical
features on eight datasets. It shows that finetuning PLMs with diffusion
degrades the reconstruction ability on OOD data. The comparison also shows that
diffusion models can effectively detect OOD samples, achieving state-of-the-art
performance in most of the datasets with an absolute accuracy improvement up to
18%. These results indicate that diffusion reduces OOD robustness of PLMs.

使用 Transformer 预训练语言模型（PLMs）中的扩散模型降低了离群值鲁棒性，并提供了对离群样本的强力检测能力。