State-of-the-art federated learning methods can perform far worse than their
centralized counterparts when clients have dissimilar data distributions. For
neural networks, even when centralized SGD easily finds a solution that is
simultaneously performant for all clients, current federated optimization
methods fail to converge to a comparable solution. We show that this
performance disparity can largely be attributed to optimization challenges
presented by nonconvexity. Specifically, we find that the early layers of the
network do learn useful features, but the final layers fail to make use of
them. That is, federated optimization applied to this non-convex problem
distorts the learning of the final layers. Leveraging this observation, we
propose a Train-Convexify-Train (TCT) procedure to sidestep this issue: first,
learn features using off-the-shelf methods (e.g., FedAvg); then, optimize a
convexified problem obtained from the network's empirical neural tangent kernel
approximation. Our technique yields accuracy improvements of up to +36% on
FMNIST and +37% on CIFAR10 when clients have dissimilar data.

提出了一种称为 Train-Convexify-Train (TCT) 的过程，以解决非凸问题中联邦优化方法失败的问题，并进一步通过 CIFAR10 和 FMNIST 数据集实现了约 37% 和 36% 的准确度改进。