Pre-training has achieved remarkable success when transferred to downstream
tasks. In machine learning, we care about not only the good performance of a
model but also its behavior under reasonable shifts of condition. The same
philosophy holds when pre-training a foundation model. However, the foundation
model may not uniformly behave well for a series of related downstream tasks.
This happens, for example, when conducting mask recovery regression where the
recovery ability or the training instances diverge like pattern features are
extracted dominantly on pre-training, but semantic features are also required
on a downstream task. This paper considers pre-training a model that guarantees
a uniformly good performance over the downstream tasks. We call this goal as
$\textit{downstream-task robustness}$. Our method first separates the upstream
task into several representative ones and applies a simple minimax loss for
pre-training. We then design an efficient algorithm to solve the minimax loss
and prove its convergence in the convex setting. In the experiments, we show
both on large-scale natural language processing and computer vision datasets
our method increases the metrics on worse-case downstream tasks. Additionally,
some theoretical explanations for why our loss is beneficial are provided.
Specifically, we show fewer samples are inherently required for the most
challenging downstream task in some cases.

本文提出了一种新的预训练模型方法，使其在从任务中保持鲁棒性，既在自然语言处理数据集上，又在计算机视觉数据集上都具有较好的表现。