Most investigations into double descent have focused on supervised models
while the few works studying self-supervised settings find a surprising lack of
the phenomenon. These results imply that double descent may not exist in
self-supervised models. We show this empirically using a standard and linear
autoencoder, two previously unstudied settings. The test loss is found to have
either a classical U-shape or to monotonically decrease instead of exhibiting a
double-descent curve. We hope that further work on this will help elucidate the
theoretical underpinnings of this phenomenon.

在自监督模型中，双下降现象可能不存在，两种未被研究的设置（标准自编码器和线性自编码器）的测试损失曲线分别为经典的 U 形或单调递减形，这一结果帮助我们更好地认识此现象的理论基础。

自我监督学习中是否存在双重跌落现象？

Does Double Descent Occur in Self-Supervised Learning?

Multi-task learning can leverage information learned by one task to benefit
the training of other tasks. Despite this capacity, naively training all tasks
together in one model often degrades performance, and exhaustively searching
through combinations of task groupings can be prohibitively expensive. As a
result, efficiently identifying the tasks that would benefit from training
together remains a challenging design question without a clear solution. In
this paper, we suggest an approach to select which tasks should train together
in multi-task learning models. Our method determines task groupings in a single
run by training all tasks together and quantifying the effect to which one
task's gradient would affect another task's loss. On the large-scale Taskonomy
computer vision dataset, we find this method can decrease test loss by 10.0%
compared to simply training all tasks together while operating 11.6 times
faster than a state-of-the-art task grouping method.

本文提出了一种方法来选择哪些任务应该在多任务学习模型中共同训练，该方法通过训练所有任务并量化一个任务的梯度对其他任务损失的影响来确定任务分组，实验结果表明相较于在一起训练所有任务，该方法可以在 11.6 倍的速度下将测试损失降低 10％。

多任务学习中高效识别任务分组

Efficiently Identifying Task Groupings for Multi-Task Learning

The generalization performance of a machine learning algorithm such as a
neural network depends in a non-trivial way on the structure of the data
distribution. To analyze the influence of data structure on test loss dynamics,
we study an exactly solveable model of stochastic gradient descent (SGD) on
mean square loss which predicts test loss when training on features with
arbitrary covariance structure. We solve the theory exactly for both Gaussian
features and arbitrary features and we show that the simpler Gaussian model
accurately predicts test loss of nonlinear random-feature models and deep
neural networks trained with SGD on real datasets such as MNIST and CIFAR-10.
We show that the optimal batch size at a fixed compute budget is typically
small and depends on the feature correlation structure, demonstrating the
computational benefits of SGD with small batch sizes. Lastly, we extend our
theory to the more usual setting of stochastic gradient descent on a fixed
subsampled training set, showing that both training and test error can be
accurately predicted in our framework on real data.

研究机器学习算法（如神经网络）的泛化能力如何受数据分布结构影响，提出了一种基于随机梯度下降的可解模型来预测任意协方差结构的特征测试误差，并在各种数据集上验证其准确性；同时证明小批量 SGD 在固定计算预算下的最优批量通常较小，取决于特征相关性结构。