Out-of-Domain (OOD) generalization is the ability of a model trained on one or more domains to generalize to unseen domains. In the ImageNet era of computer vision, evaluation sets for measuring a model's OOD performance were designed to be strictly OOD with respect to style. However, the emergence of foundation models and expansive web-scale datasets has obfuscated this evaluation process, as datasets cover a broad range of domains and risk test domain contamination. In search of the forgotten domain generalization, we create large-scale datasets subsampled from LAION -- LAION-Natural and LAION-Rendition -- that are strictly OOD to corresponding ImageNet and DomainNet test sets in terms of style. Training CLIP models on these datasets reveals that a significant portion of their performance is explained by in-domain examples. This indicates that the OOD generalization challenges from the ImageNet era still prevail and that training on web-scale data merely creates the illusion of OOD generalization. Furthermore, through a systematic exploration of combining natural and rendition datasets in varying proportions, we identify optimal mixing ratios for model generalization across these domains. Our datasets and results re-enable meaningful assessment of OOD robustness at scale -- a crucial prerequisite for improving model robustness.

本研究解决了模型在未见领域中进行泛化的挑战，尤其是在现有的大规模数据集可能导致评估过程不准确的背景下。通过创建严格与ImageNet和DomainNet测试集风格上不重合的大规模数据集，研究揭示了模型性能仍主要依赖于内部示例，这表明过去的OOD泛化问题依然存在。此外，研究确定了不同数据集组合的最佳混合比例，从而提升模型的泛化能力。 

寻找被遗忘的领域泛化