Valuable training data is often owned by independent organizations and
located in multiple data centers. Most deep learning approaches require to
centralize the multi-datacenter data for performance purpose. In practice,
however, it is often infeasible to transfer all data to a centralized data
center due to not only bandwidth limitation but also the constraints of privacy
regulations. Model averaging is a conventional choice for data parallelized
training, but its ineffectiveness is claimed by previous studies as deep neural
networks are often non-convex. In this paper, we argue that model averaging can
be effective in the decentralized environment by using two strategies, namely,
the cyclical learning rate and the increased number of epochs for local model
training. With the two strategies, we show that model averaging can provide
competitive performance in the decentralized mode compared to the
data-centralized one. In a practical environment with multiple data centers, we
conduct extensive experiments using state-of-the-art deep network architectures
on different types of data. Results demonstrate the effectiveness and
robustness of the proposed method.

本文讨论了如何在分布式环境下利用模型平均策略进行深度学习的训练，提出了采用循环学习率和增加本地模型训练轮数两种策略的方法，并在实验中证明了该方法在多个数据中心下具有竞争性的性能。