In modern decentralized applications, ensuring communication efficiency and privacy for the users are the key challenges. In order to train machine-learning models, the algorithm has to communicate to the data center and sample data for its gradient computation, thus exposing the data and increasing the communication cost. This gives rise to the need for a decentralized optimization algorithm that is communication-efficient and minimizes the number of gradient computations. To this end, we propose the primal-dual sliding with conditional gradient sliding framework, which is communication-efficient and achieves an $\varepsilon$-approximate solution with the optimal gradient complexity of $O(1/\sqrt{\varepsilon}+\sigma^2/{\varepsilon^2})$ and $O(\log(1/\varepsilon)+\sigma^2/\varepsilon)$ for the convex and strongly convex setting respectively and an LO (Linear Optimization) complexity of $O(1/\varepsilon^2)$ for both settings given a stochastic gradient oracle with variance $\sigma^2$. Compared with the prior work \cite{wai-fw-2017}, our framework relaxes the assumption of the optimal solution being a strict interior point of the feasible set and enjoys wider applicability for large-scale training using a stochastic gradient oracle. We also demonstrate the efficiency of our algorithms with various numerical experiments.

现代分散应用中，通信效率和用户隐私是关键挑战。为了训练机器学习模型，算法必须与数据中心进行通信并对其梯度计算进行采样，从而暴露数据并增加通信成本。为了解决这个问题，我们提出了一种分散优化算法，它在通信效率上高效，并通过最优梯度复杂性达到了一个ε-近似解，对于凸和强凸的设置分别为O(1/√ε+σ²/ε²)和O(log(1/ε)+σ²/ε)，对于两种设置都具有O(1/ε²)的LO(线性优化)复杂性，给定方差σ²的随机梯度预言。与之前的工作相比，我们的框架放宽了最优解是可行集的严格内点的假设，并在使用随机梯度预言进行大规模训练时具有更广泛的适用性。我们还通过各种数值实验证明了我们算法的效率。

基于梯度滑动方法的少数据神经网络分布式随机约束优化