As machine learning models are deployed ever more broadly, it becomes increasingly important that they are not only able to perform well on their training distribution, but also yield accurate predictions when confronted with distribution shift. The Distributionally Robust Optimization (DRO) framework proposes to address this issue by training models to minimize their expected risk under a collection of distributions, to imitate test-time shifts. This is most commonly achieved by instance-level re-weighting of the training objective to emulate the likelihood ratio with possible test distributions, which allows for estimating their empirical risk via importance sampling (assuming that they are subpopulations of the training distribution). However, re-weighting schemes in the literature are usually limited due to the difficulty of keeping the optimization problem tractable and the complexity of enforcing normalization constraints. In this paper, we show that three simple ideas -- mini-batch level normalization, a KL penalty and simultaneous gradient updates -- allow us to train models with DRO using a broader class of parametric likelihood ratios. In a series of experiments on both image and text classification benchmarks, we find that models trained with the resulting parametric adversaries are consistently more robust to subpopulation shifts when compared to other DRO approaches, and that the method performs reliably well with little hyper-parameter tuning. Code to reproduce our experiments can be found at https://github.com/pmichel31415/P-DRO.

本文提出了基于分布鲁棒优化的方法，通过实例加权训练机器学习模型以实现模型在测试集分布不同的情况下准确预测，并给出了通过小批量归一化、KL惩罚和梯度同步更新三个简单的想法，应用广泛的参数化似然率，训练具有分布鲁棒性的模型的案例。实验表明，与其他DRO方法相比，使用参数化对手训练的模型在子样本移位时更具鲁棒性，并且该方法在很少的超参数调整下表现出可靠的性能。

具有参数似然比的分布鲁棒模型