Real-world applications of machine learning models often confront data
distribution shifts, wherein discrepancies exist between the training and test
data distributions. In the common multi-domain multi-class setup, as the number
of classes and domains scales up, it becomes infeasible to gather training data
for every domain-class combination. This challenge naturally leads the quest
for models with Compositional Generalization (CG) ability, where models can
generalize to unseen domain-class combinations. To delve into the CG challenge,
we develop CG-Bench, a suite of CG benchmarks derived from existing real-world
image datasets, and observe that the prevalent pretraining-finetuning paradigm
on foundational models, such as CLIP and DINOv2, struggles with the challenge.
To address this challenge, we propose Compositional Feature Alignment (CFA), a
simple two-stage finetuning technique that i) learns two orthogonal linear
heads on a pretrained encoder with respect to class and domain labels, and ii)
fine-tunes the encoder with the newly learned head frozen. We theoretically and
empirically justify that CFA encourages compositional feature learning of
pretrained models. We further conduct extensive experiments on CG-Bench for
CLIP and DINOv2, two powerful pretrained vision foundation models. Experiment
results show that CFA outperforms common finetuning techniques in compositional
generalization, corroborating CFA's efficacy in compositional feature learning.

在实际的机器学习模型应用中，数据分布的变化常常导致训练数据和测试数据之间存在差异。在常见的多领域多类别设置中，随着类别和领域数量的增加，无法为每个领域 - 类别组合收集足够的训练数据。为了解决这一挑战，我们提出了一个名为 CG-Bench 的测试套件，该测试套件基于现有的真实图像数据集，发现了一种名为 Compositional Feature Alignment (CFA) 的两阶段模型微调技术，它可以促进预训练模型的组合特征学习。实验结果表明，CFA 在组合泛化方面优于常见的微调技术，证实了 CFA 在组合特征学习中的有效性。