Large neural networks pretrained on web-scale corpora are central to modern
machine learning. In this paradigm, the distribution of the large,
heterogeneous pretraining data rarely matches that of the application domain.
This work considers modifying the pretraining distribution in the case where
one has a small sample of data reflecting the targeted test conditions. We
propose an algorithm motivated by a recent formulation of this setting as an
online, bilevel optimization problem. With scalability in mind, our algorithm
prioritizes computing gradients at training points which are likely to most
improve the loss on the targeted distribution. Empirically, we show that in
some cases this approach is beneficial over existing strategies from the domain
adaptation literature but may not succeed in other cases. We propose a simple
test to evaluate when our approach can be expected to work well and point
towards further research to address current limitations.

在现代机器学习中，基于网络规模的大型神经网络学习成为核心。本研究考虑在一个与应用领域不匹配的混杂预训练数据集的情况下，通过调整预训练分布的算法来适应目标测试条件。通过在线双层优化问题的一种近期表述，提出了一种算法，并考虑了可扩展性，该算法优先计算在训练点上的梯度，这些点最有可能改善目标分布上的损失。从经验证实，在某些情况下，这种方法比现有的领域适应性策略更加有益，但在其他情况下可能不成功。我们提出了一个简单的测试来评估我们的方法何时表现良好，并指出进一步的研究来解决当前的限制。