Consider a prediction setting where a few inputs (e.g., satellite images) are expensively annotated with the prediction targets (e.g., crop types), and many inputs are cheaply annotated with auxiliary information (e.g., climate information). How should we best leverage this auxiliary information for the prediction task? Empirically across three image and time-series datasets, and theoretically in a multi-task linear regression setting, we show that (i) using auxiliary information as input features improves in-distribution error but can hurt out-of-distribution (OOD) error; while (ii) using auxiliary information as outputs of auxiliary tasks to pre-train a model improves OOD error. To get the best of both worlds, we introduce In-N-Out, which first trains a model with auxiliary inputs and uses it to pseudolabel all the in-distribution inputs, then pre-trains a model on OOD auxiliary outputs and fine-tunes this model with the pseudolabels (self-training). We show both theoretically and empirically that In-N-Out outperforms auxiliary inputs or outputs alone on both in-distribution and OOD error.

通过将辅助信息作为输入特征来提高数据内误差，再通过将其作为辅助预训练任务的输出值来提高数据外误差，提出了一种 In-N-Out 方法，该方法既可以用于伪标注所有数据内数据，也可以用于在数据内和数据外错误方面优于仅使用输入或输出辅助信息。

In-N-Out: 利用辅助信息进行预训练和自训练以提升对抗分布鲁棒性