Domain generalization aims to learn invariance across multiple training
domains, thereby enhancing generalization against out-of-distribution data.
While gradient or representation matching algorithms have achieved remarkable
success, these methods generally lack generalization guarantees or depend on
strong assumptions, leaving a gap in understanding the underlying mechanism of
distribution matching. In this work, we formulate domain generalization from a
novel probabilistic perspective, ensuring robustness while avoiding overly
conservative solutions. Through comprehensive information-theoretic analysis,
we provide key insights into the roles of gradient and representation matching
in promoting generalization. Our results reveal the complementary relationship
between these two components, indicating that existing works focusing solely on
either gradient or representation alignment are insufficient to solve the
domain generalization problem. In light of these theoretical findings, we
introduce IDM to simultaneously align the inter-domain gradients and
representations. Integrated with the proposed PDM method for complex
distribution matching, IDM achieves superior performance over various baseline
methods.

通过概率学的分析，揭示了梯度匹配和表示匹配在推广中的互补关系，并介绍了一种新的方法来同时调整跨域梯度和表示。与现有方法相比，该方法在复杂分布匹配方面表现出优越性能。

如何通过分布匹配帮助域泛化：一种信息理论分析

How Does Distribution Matching Help Domain Generalization: An  Information-theoretic Analysis

Existing feature distillation methods commonly adopt the One-to-one
Representation Matching between any pre-selected teacher-student layer pair. In
this paper, we present N-to-One Representation (NORM), a new two-stage
knowledge distillation method, which relies on a simple Feature Transform (FT)
module consisting of two linear layers. In view of preserving the intact
information learnt by the teacher network, during training, our FT module is
merely inserted after the last convolutional layer of the student network. The
first linear layer projects the student representation to a feature space
having N times feature channels than the teacher representation from the last
convolutional layer, and the second linear layer contracts the expanded output
back to the original feature space. By sequentially splitting the expanded
student representation into N non-overlapping feature segments having the same
number of feature channels as the teacher's, they can be readily forced to
approximate the intact teacher representation simultaneously, formulating a
novel many-to-one representation matching mechanism conditioned on a single
teacher-student layer pair. After training, such an FT module will be naturally
merged into the subsequent fully connected layer thanks to its linear property,
introducing no extra parameters or architectural modifications to the student
network at inference. Extensive experiments on different visual recognition
benchmarks demonstrate the leading performance of our method. For instance, the
ResNet18|MobileNet|ResNet50-1/4 model trained by NORM reaches
72.14%|74.26%|68.03% top-1 accuracy on the ImageNet dataset when using a
pre-trained ResNet34|ResNet50|ResNet50 model as the teacher, achieving an
absolute improvement of 2.01%|4.63%|3.03% against the individually trained
counterpart. Code is available at this https URL

本研究提出了 N-to-One Representation Matching (NORM) 知识蒸馏方法，其主要利用由两个线性层组成的 Feature Transform (FT) 模块，可以在训练中保留教师网络学习到的信息，并且通过新颖的多对一表示匹配机制，推动学生网络逼近教师网络的表示。 在不引入额外参数或结构修改的情况下，在多个视觉识别基准测试中展示了该方法的领先性能。

NORM: 基于 N 对一表示匹配的知识蒸馏

NORM: Knowledge Distillation via N-to-One Representation Matching

Federated learning is a distributed, privacy-aware learning scenario which
trains a single model on data belonging to several clients. Each client trains
a local model on its data and the local models are then aggregated by a central
party. Current federated learning methods struggle in cases with heterogeneous
client-side data distributions which can quickly lead to divergent local models
and a collapse in performance. Careful hyper-parameter tuning is particularly
important in these cases but traditional automated hyper-parameter tuning
methods would require several training trials which is often impractical in a
federated learning setting. We describe a two-pronged solution to the issues of
robustness and hyper-parameter tuning in federated learning settings. We
propose a novel representation matching scheme that reduces the divergence of
local models by ensuring the feature representations in the global (aggregate)
model can be derived from the locally learned representations. We also propose
an online hyper-parameter tuning scheme which uses an online version of the
REINFORCE algorithm to find a hyper-parameter distribution that maximizes the
expected improvements in training loss. We show on several benchmarks that our
two-part scheme of local representation matching and global adaptive
hyper-parameters significantly improves performance and training robustness.

本文提出了一种解决分散式学习中数据分布不一致、超参数调整等问题的两种方案，分别是特征匹配方案和自适应超参数调整方案，并在多个基准测试中证明，这两种方案可以显著提高性能和训练鲁棒性。