Loss of plasticity is a phenomenon where neural networks become more
difficult to train during the course of learning. Continual learning algorithms
seek to mitigate this effect by sustaining good predictive performance while
maintaining network trainability. We develop new techniques for improving
continual learning by first reconsidering how initialization can ensure
trainability during early phases of learning. From this perspective, we derive
new regularization strategies for continual learning that ensure beneficial
initialization properties are better maintained throughout training. In
particular, we investigate two new regularization techniques for continual
learning: (i) Wasserstein regularization toward the initial weight
distribution, which is less restrictive than regularizing toward initial
weights; and (ii) regularizing weight matrix singular values, which directly
ensures gradient diversity is maintained throughout training. We present an
experimental analysis that shows these alternative regularizers can improve
continual learning performance across a range of supervised learning tasks and
model architectures. The alternative regularizers prove to be less sensitive to
hyperparameters while demonstrating better training in individual tasks,
sustaining trainability as new tasks arrive, and achieving better
generalization performance.

神经网络的塑性缺失意味着在学习过程中网络训练变得更加困难。我们通过改进初始化技术、提出新的正则化策略以及保持梯度多样性来改善持续学习，并证明这些替代技术在不同的监督学习任务和模型结构上能够提高持续学习性能。

通过谱正则化实现持续学习

Learning Continually by Spectral Regularization

Federated learning refers to a distributed machine learning paradigm in which
data samples are decentralized and distributed among multiple clients. These
samples may exhibit statistical heterogeneity, which refers to data
distributions are not independent and identical across clients. Additionally,
system heterogeneity, or variations in the computational power of the clients,
introduces biases into federated learning. The combined effects of statistical
and system heterogeneity can significantly reduce the efficiency of federated
optimization. However, the impact of hybrid heterogeneity is not rigorously
discussed. This paper explores how hybrid heterogeneity affects federated
optimization by investigating server-side optimization. The theoretical results
indicate that adaptively maximizing gradient diversity in server update
direction can help mitigate the potential negative consequences of hybrid
heterogeneity. To this end, we introduce a novel server-side gradient-based
optimizer \textsc{FedAWARE} with theoretical guarantees provided. Intensive
experiments in heterogeneous federated settings demonstrate that our proposed
optimizer can significantly enhance the performance of federated learning
across varying degrees of hybrid heterogeneity.

研究了混合异构性如何影响联邦优化，通过调查服务器端优化表明，自适应地最大化梯度多样性在服务器更新方向上可以帮助缓解混合异构性的潜在负面影响，引入了具有理论保证的新型基于梯度的优化器 FedAWARE，通过在异构联邦设置中进行大量实验表明，我们的提议优化器可以显著提升不同程度的混合异构性下联邦学习的性能。

通过梯度多样性最大化解决联邦优化的混合异质性问题

Tackling Hybrid Heterogeneity on Federated Optimization via Gradient  Diversity Maximization

In this work, we propose GPT-FL, a generative pre-trained model-assisted
federated learning (FL) framework. At its core, GPT-FL leverages generative
pre-trained models to generate diversified synthetic data. These generated data
are used to train a downstream model on the server, which is then fine-tuned
with private client data under the standard FL framework. We show that GPT-FL
consistently outperforms state-of-the-art FL methods in terms of model test
accuracy, communication efficiency, and client sampling efficiency. Through
comprehensive ablation analysis, we discover that the downstream model
generated by synthetic data plays a crucial role in controlling the direction
of gradient diversity during FL training, which enhances convergence speed and
contributes to the notable accuracy boost observed with GPT-FL. Also,
regardless of whether the target data falls within or outside the domain of the
pre-trained generative model, GPT-FL consistently achieves significant
performance gains, surpassing the results obtained by models trained solely
with FL or synthetic data.

本文提出了一个基于生成式预训练模型辅助联邦学习框架的 GPT-FL 方法，在提高通信效率和客户抽样效率的同时，优化梯度多样性来增强模型收敛速度和提高模型测试准确性。