We study a mismatch between the deep learning recommendation models' flat architecture, common distributed training paradigm and hierarchical data center topology. To address the associated inefficiencies, we propose Disaggregated Multi-Tower (DMT), a modeling technique that consists of (1) Semantic-preserving Tower Transform (SPTT), a novel training paradigm that decomposes the monolithic global embedding lookup process into disjoint towers to exploit data center locality; (2) Tower Module (TM), a synergistic dense component attached to each tower to reduce model complexity and communication volume through hierarchical feature interaction; and (3) Tower Partitioner (TP), a feature partitioner to systematically create towers with meaningful feature interactions and load balanced assignments to preserve model quality and training throughput via learned embeddings. We show that DMT can achieve up to 1.9x speedup compared to the state-of-the-art baselines without losing accuracy across multiple generations of hardware at large data center scales.

研究了深度学习推荐模型的平面结构、常见的分布式训练范式和层次化数据中心拓扑之间的不匹配性，提出了分离式多塔（DMT）模型技术，包括保持语义的塔变换（SPTT）、附加在每个塔上的塔模块（TM）以减少模型复杂性和通信量以及可系统创建带有有意义特征交互且负载平衡分配以保持模型质量和训练吞吐量的塔分区器（TP）。

分立多塔：拓扑感知建模技术用于高效大规模推荐