Stochastic Gradient Descent (SGD) is a popular algorithm that can achieve
state-of-the-art performance on a variety of machine learning tasks. Several
researchers have recently proposed schemes to parallelize SGD, but all require
performance-destroying memory locking and synchronization. This work aims to
show using novel theoretical analysis, algorithms, and implementation that SGD
can be implemented without any locking. We present an update scheme called
HOGWILD! which allows processors access to shared memory with the possibility
of overwriting each other's work. We show that when the associated optimization
problem is sparse, meaning most gradient updates only modify small parts of the
decision variable, then HOGWILD! achieves a nearly optimal rate of convergence.
We demonstrate experimentally that HOGWILD! outperforms alternative schemes
that use locking by an order of magnitude.

本文研究的主要问题是如何并行化 Stochastic Gradient Descent 算法，提出了一种称为 HOGWILD！的更新策略，可以在没有锁定的情况下实现并行计算，并展示了该策略在稀疏优化问题上接近最优的收敛率。实验证明 HOGWILD！的表现比使用锁的并行化方案要高出一个数量级。