Modern advancements in large-scale machine learning would be impossible
without the paradigm of data-parallel distributed computing. Since distributed
computing with large-scale models imparts excessive pressure on communication
channels, significant recent research has been directed toward co-designing
communication compression strategies and training algorithms with the goal of
reducing communication costs. While pure data parallelism allows better data
scaling, it suffers from poor model scaling properties. Indeed, compute nodes
are severely limited by memory constraints, preventing further increases in
model size. For this reason, the latest achievements in training giant neural
network models also rely on some form of model parallelism. In this work, we
take a closer theoretical look at Independent Subnetwork Training (IST), which
is a recently proposed and highly effective technique for solving the
aforementioned problems. We identify fundamental differences between IST and
alternative approaches, such as distributed methods with compressed
communication, and provide a precise analysis of its optimization performance
on a quadratic model.

本文介绍了一种名为 Independent Subnetwork Training（IST）的高效神经网络模型并行化技术，其与压缩通信的分布式方法有着基本差异，在解决分布式大规模神经网络模型训练中的通信与计算问题上具有良好的优化性能。