Federated learning is an important privacy-preserving multi-party learning
paradigm, involving collaborative learning with others and local updating on
private data. Model heterogeneity and catastrophic forgetting are two crucial
challenges, which greatly limit the applicability and generalizability. This
paper presents a novel FCCL+, federated correlation and similarity learning
with non-target distillation, facilitating the both intra-domain
discriminability and inter-domain generalization. For heterogeneity issue, we
leverage irrelevant unlabeled public data for communication between the
heterogeneous participants. We construct cross-correlation matrix and align
instance similarity distribution on both logits and feature levels, which
effectively overcomes the communication barrier and improves the generalizable
ability. For catastrophic forgetting in local updating stage, FCCL+ introduces
Federated Non Target Distillation, which retains inter-domain knowledge while
avoiding the optimization conflict issue, fulling distilling privileged
inter-domain information through depicting posterior classes relation.
Considering that there is no standard benchmark for evaluating existing
heterogeneous federated learning under the same setting, we present a
comprehensive benchmark with extensive representative methods under four domain
shift scenarios, supporting both heterogeneous and homogeneous federated
settings. Empirical results demonstrate the superiority of our method and the
efficiency of modules on various scenarios.

本文提出了一种新的基于 FCCL + 的联邦相关性和相似性学习方法，通过非目标蒸馏，在异构参与方之间利用不相关的未标记公共数据进行通信，构建了交叉相关矩阵并在逻辑和特征层面上对实例相似度分布进行对齐，旨在解决模型异质性和灾难性遗忘问题，并为此提出了一种综合评估标准，实验证明了该方法在各种场景下的优越性和模块的高效性。