In this work, we propose a federated dynamical low-rank training (FeDLRT)
scheme to reduce client compute and communication costs - two significant
performance bottlenecks in horizontal federated learning. Our method builds
upon dynamical low-rank splitting schemes for manifold-constrained optimization
to create a global low-rank basis of network weights, which enables client
training on a small coefficient matrix. A consistent global low-rank basis
allows us to incorporate a variance correction scheme and prove global loss
descent and convergence to a stationary point. Dynamic augmentation and
truncation of the low-rank bases automatically optimizes computing and
communication resource utilization. We demonstrate the efficiency of FeDLRT in
an array of computer vision benchmarks and show a reduction of client compute
and communication costs by up to an order of magnitude with minimal impacts on
global accuracy.

我们提出了一种联邦动态低秩训练 (FeDLRT) 方案，用于减少水平联邦学习中的客户端计算和通信成本，并在计算机视觉基准测试中证明了 FeDLRT 的效率。

具有全局损失收敛保证的联邦动态低秩训练

Federated Dynamical Low-Rank Training with Global Loss Convergence  Guarantees

We expect the generalization error to improve with more samples from a
similar task, and to deteriorate with more samples from an out-of-distribution
(OOD) task. In this work, we show a counter-intuitive phenomenon: the
generalization error of a task can be a non-monotonic function of the number of
OOD samples. As the number of OOD samples increases, the generalization error
on the target task improves before deteriorating beyond a threshold. In other
words, there is value in training on small amounts of OOD data. We use Fisher's
Linear Discriminant on synthetic datasets and deep networks on computer vision
benchmarks such as MNIST, CIFAR-10, CINIC-10, PACS and DomainNet to demonstrate
and analyze this phenomenon. In the idealistic setting where we know which
samples are OOD, we show that these non-monotonic trends can be exploited using
an appropriately weighted objective of the target and OOD empirical risk. While
its practical utility is limited, this does suggest that if we can detect OOD
samples, then there may be ways to benefit from them. When we do not know which
samples are OOD, we show how a number of go-to strategies such as
data-augmentation, hyper-parameter optimization, and pre-training are not
enough to ensure that the target generalization error does not deteriorate with
the number of OOD samples in the dataset.

在本研究中，我们发现了一个反直觉的现象：在涉及目标任务的样本数量增加之前，由于少量的来自未知分布数据的样本，可以提高任务的泛化性能，但随着样本数量的增加，泛化误差达到阈值后会逐渐下降；我们采用合成数据集上的 Fisher's Linear Discriminant 和计算机视觉基准数据集（如 MNIST、CIFAR-10、CINIC-10、PACS 和 DomainNet）上的深度神经网络来证明这一现象；在我们知道哪些样本是未知分布的理想情况下，我们可以使用适当加权的目标和外部风险的目标函数来利用这些非单调趋势，但其实际效用有限，此外，当我们不知道哪些样本是未知分布时，数据增强、超参数优化和预训练等常用策略仍然无法保证目标泛化误差不会随着未知分布样本数量的增加而下降。