It is impossible today to pretend that the practice of machine learning is
compatible with the idea that training and testing data follow the same
distribution. Several authors have recently used ensemble techniques to show
how scenarios involving multiple data distributions are best served by
representations that are both richer than those obtained by regularizing for
the best in-distribution performance, and richer than those obtained under the
influence of the implicit sparsity bias of common stochastic gradient
procedures.
This contribution investigates the use of very high dropout rates instead of
ensembles to obtain such rich representations. Although training a deep network
from scratch using such dropout rates is virtually impossible, fine-tuning a
large pre-trained model under such conditions is not only possible but also
achieves out-of-distribution performances that exceed those of both ensembles
and weight averaging methods such as model soups. This result has practical
significance because the importance of the fine-tuning scenario has
considerably grown in recent years. This result also provides interesting
insights on the nature of rich representations and on the intrinsically linear
nature of fine-tuning a large network using a comparatively small dataset.

使用高辍学率而不是集成技术，通过精细调整来获得丰富的表示，其实现出超过集成方法和权重平均方法的超越分布性能，从而为庞大预训练模型的微调情景提供了实际意义。

利用非常大的 Dropout 进行微调

Fine-tuning with Very Large Dropout

Graduation and dropout rates have always been a serious consideration for
educational institutions and students. High dropout rates negatively impact
both the lives of individual students and institutions. To address this
problem, this study examined university dropout prediction using academic,
demographic, socioeconomic, and macroeconomic data types. Additionally, we
performed associated factor analysis to analyze which type of data would be
most influential on the performance of machine learning models in predicting
graduation and dropout status. These features were used to train four binary
classifiers to determine if students would graduate or drop out. The overall
performance of the classifiers in predicting dropout status had an average
ROC-AUC score of 0.935. The data type most influential to the model performance
was found to be academic data, with the average ROC-AUC score dropping from
0.935 to 0.811 when excluding all academic-related features from the data set.
Preliminary results indicate that a correlation does exist between data types
and dropout status.

大学辍学预测使用学术、人口统计、社会经济和宏观经济数据类型，通过训练四个二元分类器来预测学生是否会毕业或辍学，结果发现学术数据类型对模型性能最有影响。

大学生退学原因预测及相关因素分析的机器学习技术应用

Why Do Students Drop Out? University Dropout Prediction and Associated  Factor Analysis Using Machine Learning Techniques

We explore a recently proposed Variational Dropout technique that provided an
elegant Bayesian interpretation to Gaussian Dropout. We extend Variational
Dropout to the case when dropout rates are unbounded, propose a way to reduce
the variance of the gradient estimator and report first experimental results
with individual dropout rates per weight. Interestingly, it leads to extremely
sparse solutions both in fully-connected and convolutional layers. This effect
is similar to automatic relevance determination effect in empirical Bayes but
has a number of advantages. We reduce the number of parameters up to 280 times
on LeNet architectures and up to 68 times on VGG-like networks with a
negligible decrease of accuracy.

本研究采用变分丢失技术，提供了一种优雅的高斯丢失的贝叶斯解释，将其扩展到丢失速率无界的情况，提出一种减少梯度估计器方差的方法，并在每个权重的情况下报告第一个实验结果。有趣的是，在完全连接和卷积层中都导致极度稀疏的解决方案。这种效应类似于实证贝叶斯中的自动相关确定效应，但具有许多优势。我们在 LeNet 架构上将参数减少了最多 280 倍，并在类似 VGG 的网络上将参数减少了最多 68 倍，同时准确度几乎不减。