Empirical risk minimization (ERM) is known in practice to be non-robust to
distributional shift where the training and the test distributions are
different. A suite of approaches, such as importance weighting, and variants of
distributionally robust optimization (DRO), have been proposed to solve this
problem. But a line of recent work has empirically shown that these approaches
do not significantly improve over ERM in real applications with distribution
shift. The goal of this work is to obtain a comprehensive theoretical
understanding of this intriguing phenomenon. We first posit the class of
Generalized Reweighting (GRW) algorithms, as a broad category of approaches
that iteratively update model parameters based on iterative reweighting of the
training samples. We show that when overparameterized models are trained under
GRW, the resulting models are close to that obtained by ERM. We also show that
adding small regularization which does not greatly affect the empirical
training accuracy does not help. Together, our results show that a broad
category of what we term GRW approaches are not able to achieve
distributionally robust generalization. Our work thus has the following
sobering takeaway: to make progress towards distributionally robust
generalization, we either have to develop non-GRW approaches, or perhaps devise
novel classification/regression loss functions that are adapted to the class of
GRW approaches.

本文介绍了一种名为广义重加权算法（GRW）的类别，它通过迭代地重新加权训练样本来更新模型参数。我们发现在采用 GRW 算法的过拟合模型下，所得到的模型与采用 Empirical risk minimization 得到的模型非常相似。此外，在 GRW 算法不使用小规则化方法的情况下，得到的结果也是不尽如人意的，我们需要更多的探索与研究。