Distributed machine learning (ML) can bring more computational resources to
bear than single-machine learning, thus enabling reductions in training time.
Distributed learning partitions models and data over many machines, allowing
model and dataset sizes beyond the available compute power and memory of a
single machine. In practice though, distributed ML is challenging when
distribution is mandatory, rather than chosen by the practitioner. In such
scenarios, data could unavoidably be separated among workers due to limited
memory capacity per worker or even because of data privacy issues. There,
existing distributed methods will utterly fail due to dominant transfer costs
across workers, or do not even apply.
We propose a new approach to distributed fully connected neural network
learning, called independent subnet training (IST), to handle these cases. In
IST, the original network is decomposed into a set of narrow subnetworks with
the same depth. These subnetworks are then trained locally before parameters
are exchanged to produce new subnets and the training cycle repeats. Such a
naturally "model parallel" approach limits memory usage by storing only a
portion of network parameters on each device. Additionally, no requirements
exist for sharing data between workers (i.e., subnet training is local and
independent) and communication volume and frequency are reduced by decomposing
the original network into independent subnets. These properties of IST can cope
with issues due to distributed data, slow interconnects, or limited device
memory, making IST a suitable approach for cases of mandatory distribution. We
show experimentally that IST results in training times that are much lower than
common distributed learning approaches.

提出了一种名为独立子网训练（IST）的分布式全连接神经网络学习新方法，通过将原始网络分解成具有相同深度的一组窄子网络，使 IST 能够有效处理强制分布式数据、慢速互连或有限设备内存等问题，并且训练时间远低于传统的分布式学习方法。