We introduce equi-tuning, a novel fine-tuning method that transforms (potentially non-equivariant) pretrained models into group equivariant models while incurring minimum $L_2$ loss between the feature representations of the pretrained and the equivariant models. Large pretrained models can be equi-tuned for different groups to satisfy the needs of various downstream tasks. Equi-tuned models benefit from both group equivariance as an inductive bias and semantic priors from pretrained models. We provide applications of equi-tuning on three different tasks: image classification, compositional generalization in language, and fairness in natural language generation (NLG). We also provide a novel group-theoretic definition for fairness in NLG. The effectiveness of this definition is shown by testing it against a standard empirical method of fairness in NLG. We provide experimental results for equi-tuning using a variety of pretrained models: Alexnet, Resnet, VGG, and Densenet for image classification; RNNs, GRUs, and LSTMs for compositional generalization; and GPT2 for fairness in NLG. We test these models on benchmark datasets across all considered tasks to show the generality and effectiveness of the proposed method.

本文介绍了 equi-tuning，一种新颖的微调方法，将（潜在非等变的）预训练模型转换为群等变模型，并在预训练和等变模型的特征表示之间产生最小的 L2 损失。我们提出了 equi-tuning 的应用，如图像分类、语言的组成泛化以及自然语言生成的公平性，并为 NLG 中的公平性提供了一种新的群论定义，这种定义的有效性通过测试它与标准实证方法的 NLG 中的公平性相比得到证明。

Equi-Tuning: 预训练模型的群等变微调