This paper advocates a new paradigm Personalized Empirical Risk Minimization
(PERM) to facilitate learning from heterogeneous data sources without imposing
stringent constraints on computational resources shared by participating
devices. In PERM, we aim to learn a distinct model for each client by learning
who to learn with and personalizing the aggregation of local empirical losses
by effectively estimating the statistical discrepancy among data distributions,
which entails optimal statistical accuracy for all local distributions and
overcomes the data heterogeneity issue. To learn personalized models at scale,
we propose a distributed algorithm that replaces the standard model averaging
with model shuffling to simultaneously optimize PERM objectives for all
devices. This also allows us to learn distinct model architectures (e.g.,
neural networks with different numbers of parameters) for different clients,
thus confining underlying memory and compute resources of individual clients.
We rigorously analyze the convergence of the proposed algorithm and conduct
experiments that corroborate the effectiveness of the proposed paradigm.

该论文提出了一种新的个性化经验风险最小化（PERM）范式，以便从异质数据源中学习，而不对参与设备共享的计算资源施加严格的约束。在 PERM 中，我们旨在通过有效估计数据分布之间的统计差异来学习每个客户端的不同模型，从而实现所有局部分布的最佳统计准确度并克服数据异质性问题。为了在规模上学习个性化模型，我们提出了一种分布式算法，用模型洗牌来同时优化所有设备的 PERM 目标，取代了标准的模型平均。这还允许我们为不同的客户端学习不同的模型架构（例如，具有不同参数数量的神经网络），从而限制了各个客户端的底层存储和计算资源。我们对所提算法的收敛性进行了严格分析，并进行了实验证明了所提范式的有效性。