We present Distributed Equivalent Substitution (DES) training, a novel distributed training framework for recommender systems with large-scale dynamic sparse features. Our framework achieves faster convergence with less communication overhead and better computing resource utilization. DES strategy splits a weights-rich operator into sub-operators with co-located weights and aggregates partial results with much smaller communication cost to form a computationally equivalent substitution to the original operator. We show that for different types of models that recommender systems use, we can always find computational equivalent substitutions and splitting strategies for their weights-rich operators with theoretical communication load reduced ranging from 72.26% to 99.77%. We also present an implementation of DES that outperforms state-of-the-art recommender systems. Experiments show that our framework achieves up to 83% communication savings compared to other recommender systems, and can bring up to 4.5x improvement on throughput for deep models.

本文介绍了一种新的分布式训练框架 Distributed Equivalent Substitution（DES）进行大规模推荐系统的训练，采用同步训练的方式，采用子操作替代了权重操作，在结果聚合方面也采用聚合部分结果代替直接传输稀疏权重的方式，通过在工业场景下的多个流行的深度学习推荐模型中成功应用 DES 训练，展示了比其他基于 PS 的推荐系统更高的 AUC 值和更高的吞吐量，并实现了高达 68.7% 的通信开销节约。

大规模推荐系统的分布式等效替换训练