Topographic feature maps are low dimensional representations of data, that
preserve spatial dependencies. Current methods of training such maps (e.g. self
organizing maps - SOM, generative topographic maps) require centralized control
and synchronous execution, which restricts scalability. We present an algorithm
that uses $N$ autonomous units to generate a feature map by distributed
asynchronous training. Unit autonomy is achieved by sparse interaction in time
\& space through the combination of a distributed heuristic search, and a
cascade-driven weight updating scheme governed by two rules: a unit i) adapts
when it receives either a sample, or the weight vector of a neighbor, and ii)
broadcasts its weight vector to its neighbors after adapting for a predefined
number of times. Thus, a vector update can trigger an avalanche of adaptation.
We map avalanching to a statistical mechanics model, which allows us to
parametrize the statistical properties of cascading. Using MNIST, we
empirically investigate the effect of the heuristic search accuracy and the
cascade parameters on map quality. We also provide empirical evidence that
algorithm complexity scales at most linearly with system size $N$. The proposed
approach is found to perform comparably with similar methods in classification
tasks across multiple datasets.

提出一种基于自主单元、分布式异步训练、电路驱动的权重更新方案以及启发式搜索的算法，使得拓扑特征图的训练在无需中央控制和同步执行的条件下实现，可进行可扩展的大规模数据集处理，同时实验结果表明算法在多个数据集上的分类任务中性能表现稳定。

异步训练的分布式拓扑地图

Asynchronously Trained Distributed Topographic Maps

Distributed asynchronous offline training has received widespread attention
in recent years because of its high performance on large-scale data and complex
models. As data are distributed from cloud-centric to edge nodes, a big
challenge for distributed machine learning systems is how to handle native and
natural non-independent and identically distributed (non-IID) data for
training. Previous asynchronous training methods do not have a satisfying
performance on non-IID data because it would result in that the training
process fluctuates greatly which leads to an abnormal convergence. We propose a
gradient scheduling algorithm with partly averaged gradients and global
momentum (GSGM) for non-IID data distributed asynchronous training. Our key
idea is to apply global momentum and local average to the biased gradient after
scheduling, in order to make the training process steady. Experimental results
show that for non-IID data training under the same experimental conditions,
GSGM on popular optimization algorithms can achieve a 20% increase in training
stability with a slight improvement in accuracy on Fashion-Mnist and CIFAR-10
datasets. Meanwhile, when expanding distributed scale on CIFAR-100 dataset that
results in sparse data distribution, GSGM can perform a 37% improvement on
training stability. Moreover, only GSGM can converge well when the number of
computing nodes grows to 30, compared to the state-of-the-art distributed
asynchronous algorithms. At the same time, GSGM is robust to different degrees
of non-IID data.

提出了一种利用全局动量和局部平均梯度的梯度调度算法 GSGM 来处理非 IID 数据的分布式异步训练问题，并进行了一系列实验，结果表明 GSGM 算法相对于同类算法可以提高 20% 的训练稳定性并略微提高准确性，同时在稀疏数据分布的情况下可以提高 37% 的训练稳定性，在 30 个计算节点的情况下仍表现出较好的收敛性能，对不同程度的非 IID 数据鲁棒性强。