Federated learning encounters a critical challenge of data heterogeneity,
adversely affecting the performance and convergence of the federated model.
Various approaches have been proposed to address this issue, yet their
effectiveness is still limited. Recent studies have revealed that the federated
model suffers severe forgetting in local training, leading to global forgetting
and performance degradation. Although the analysis provides valuable insights,
a comprehensive understanding of the vulnerable classes and their impact
factors is yet to be established. In this paper, we aim to bridge this gap by
systematically analyzing the forgetting degree of each class during local
training across different communication rounds. Our observations are: (1) Both
missing and non-dominant classes suffer similar severe forgetting during local
training, while dominant classes show improvement in performance. (2) When
dynamically reducing the sample size of a dominant class, catastrophic
forgetting occurs abruptly when the proportion of its samples is below a
certain threshold, indicating that the local model struggles to leverage a few
samples of a specific class effectively to prevent forgetting. Motivated by
these findings, we propose a novel and straightforward algorithm called
Federated Knowledge Anchor (FedKA). Assuming that all clients have a single
shared sample for each class, the knowledge anchor is constructed before each
local training stage by extracting shared samples for missing classes and
randomly selecting one sample per class for non-dominant classes. The knowledge
anchor is then utilized to correct the gradient of each mini-batch towards the
direction of preserving the knowledge of the missing and non-dominant classes.
Extensive experimental results demonstrate that our proposed FedKA achieves
fast and stable convergence, significantly improving accuracy on popular
benchmarks.

在本文中我们通过系统分析每个类别在不同通信轮次中的本地训练中的遗忘程度，填补了对易受损类别及其影响因素的全面认识的空白。我们的观察是：(1) 不见类别和非主导类别在本地训练中都遭受到了相似的严重遗忘，而主导类别的性能有所改善。(2) 当动态减少一个主导类别的样本数量时，当其样本比例低于一定阈值时会突然发生灾难性遗忘，这表明本地模型在有效利用特定类别的少量样本以防止遗忘方面存在困难。受到这些发现的启发，我们提出了一种新颖且直接的算法称为 Federated Knowledge Anchor (FedKA)。在每个本地训练阶段之前，我们通过提取缺失类别的共享样本和随机选择每个非主导类别的一个样本构建知识锚。然后利用知识锚来校正每个小批量的梯度，以保留缺失和非主导类别的知识。大量的实验结果表明，我们提出的 FedKA 算法实现了快速稳定的收敛，在流行的基准测试中显著提高了准确性。

面向快速稳定的联邦学习：通过知识锚实现异构性对抗

Towards Fast and Stable Federated Learning: Confronting Heterogeneity  via Knowledge Anchor

Adversarial training is one of the most effective approaches to improve model
robustness against adversarial examples. However, previous works mainly focus
on the overall robustness of the model, and the in-depth analysis on the role
of each class involved in adversarial training is still missing. In this paper,
we propose to analyze the class-wise robustness in adversarial training. First,
we provide a detailed diagnosis of adversarial training on six benchmark
datasets, i.e., MNIST, CIFAR-10, CIFAR-100, SVHN, STL-10 and ImageNet.
Surprisingly, we find that there are remarkable robustness discrepancies among
classes, leading to unbalance/unfair class-wise robustness in the robust
models. Furthermore, we keep investigating the relations between classes and
find that the unbalanced class-wise robustness is pretty consistent among
different attack and defense methods. Moreover, we observe that the stronger
attack methods in adversarial learning achieve performance improvement mainly
from a more successful attack on the vulnerable classes (i.e., classes with
less robustness). Inspired by these interesting findings, we design a simple
but effective attack method based on the traditional PGD attack, named
Temperature-PGD attack, which proposes to enlarge the robustness disparity
among classes with a temperature factor on the confidence distribution of each
image. Experiments demonstrate our method can achieve a higher attack rate than
the PGD attack. Furthermore, from the defense perspective, we also make some
modifications in the training and inference phase to improve the robustness of
the most vulnerable class, so as to mitigate the large difference in class-wise
robustness. We believe our work can contribute to a more comprehensive
understanding of adversarial training as well as rethinking the class-wise
properties in robust models.

本文研究了对抗训练中的类别鲁棒性问题，提出了一种基于温度的 PGD 攻击方法，并对训练和推断阶段进行了改进，以降低类别鲁棒性差异。实验结果表明，该方法可以取得更高的攻击成功率。