Federated learning (FedL) has emerged as a popular technique for distributing
model training over a set of wireless devices, via iterative local updates (at
devices) and global aggregations (at the server). In this paper, we develop
parallel successive learning (PSL), which expands the FedL architecture along
three dimensions: (i) Network, allowing decentralized cooperation among the
devices via device-to-device (D2D) communications. (ii) Heterogeneity,
interpreted at three levels: (ii-a) Learning: PSL considers heterogeneous
number of stochastic gradient descent iterations with different mini-batch
sizes at the devices; (ii-b) Data: PSL presumes a dynamic environment with data
arrival and departure, where the distributions of local datasets evolve over
time, captured via a new metric for model/concept drift. (ii-c) Device: PSL
considers devices with different computation and communication capabilities.
(iii) Proximity, where devices have different distances to each other and the
access point. PSL considers the realistic scenario where global aggregations
are conducted with idle times in-between them for resource efficiency
improvements, and incorporates data dispersion and model dispersion with local
model condensation into FedL. Our analysis sheds light on the notion of cold
vs. warmed up models, and model inertia in distributed machine learning. We
then propose network-aware dynamic model tracking to optimize the model
learning vs. resource efficiency tradeoff, which we show is an NP-hard
signomial programming problem. We finally solve this problem through proposing
a general optimization solver. Our numerical results reveal new findings on the
interdependencies between the idle times in-between the global aggregations,
model/concept drift, and D2D cooperation configuration.

本文对联邦学习进行了拓展，提出了并行连续学习 (PSL)，利用 PSL 的框架，实现了从三个方面的拓展：网络（通过设备之间的设备对设备通信进行分散合作），异构性（考虑到训练设备之间的数量、数据来源和性能差异），以及接近度（设备之间不同的距离和不同的访问点）。PSL 实现了模型的分散和本地模型凝聚作为联邦学习方法的一部分，并提出了网络感知动态模型跟踪以优化模型学习和资源效率之间的权衡。