Modern data-driven and distributed learning frameworks deal with diverse
massive data generated by clients spread across heterogeneous environments.
Indeed, data heterogeneity is a major bottleneck in scaling up many distributed
learning paradigms. In many settings however, heterogeneous data may be
generated in clusters with shared structures, as is the case in several
applications such as federated learning where a common latent variable governs
the distribution of all the samples generated by a client. It is therefore
natural to ask how the underlying clustered structures in distributed data can
be exploited to improve learning schemes. In this paper, we tackle this
question in the special case of estimating $d$-dimensional parameters of a
two-component mixture of linear regressions problem where each of $m$ nodes
generates $n$ samples with a shared latent variable. We employ the well-known
Expectation-Maximization (EM) method to estimate the maximum likelihood
parameters from $m$ batches of dependent samples each containing $n$
measurements. Discarding the clustered structure in the mixture model, EM is
known to require $O(\log(mn/d))$ iterations to reach the statistical accuracy
of $O(\sqrt{d/(mn)})$. In contrast, we show that if initialized properly, EM on
the structured data requires only $O(1)$ iterations to reach the same
statistical accuracy, as long as $m$ grows up as $e^{o(n)}$. Our analysis
establishes and combines novel asymptotic optimization and generalization
guarantees for population and empirical EM with dependent samples, which may be
of independent interest.

在分布式数据中，通过结构化数据进行 EM 估计的集群结构可有效提高学习方案，最多需要 O (1) 次迭代以达到相同的统计准确性，只要 m 按照 e^{o (n)} 增长。