Low-rank model compression is a widely used technique for reducing the
computational load when training machine learning models. However, existing
methods often rely on relaxing the low-rank constraint of the model weights
using a regularized nuclear norm penalty, which requires an appropriate
hyperparameter that can be difficult to determine in practice. Furthermore,
existing compression techniques are not directly applicable to efficient
over-the-air (OTA) aggregation in federated learning (FL) systems for
distributed Internet-of-Things (IoT) scenarios. In this paper, we propose a
novel manifold optimization formulation for low-rank model compression in FL
that does not relax the low-rank constraint. Our optimization is conducted
directly over the low-rank manifold, guaranteeing that the model is exactly
low-rank. We also introduce a consensus penalty in the optimization formulation
to support OTA aggregation. Based on our optimization formulation, we propose
an alternating Riemannian optimization algorithm with a precoder that enables
efficient OTA aggregation of low-rank local models without sacrificing training
performance. Additionally, we provide convergence analysis in terms of key
system parameters and conduct extensive experiments with real-world datasets to
demonstrate the effectiveness of our proposed Riemannian low-rank model
compression scheme compared to various state-of-the-art baselines.

本文提出了一种新颖的低秩模型压缩的流形优化公式，其不需要放松低秩约束，并介绍了一种共识惩罚来支持 OTA 聚合。基于我们的优化公式，我们提出了一种交替黎曼优化算法，具有有效的 OTA 聚合，同时不会影响训练性能。在真实世界的数据集方面进行了实验，在各个最先进的基线上展示了我们提出的黎曼低秩模型压缩方案的有效性。