In practical distributed systems, workers are typically not homogeneous, and
due to differences in hardware configurations and network conditions, can have
highly varying processing times. We consider smooth nonconvex finite-sum
(empirical risk minimization) problems in this setup and introduce a new
parallel method, Freya PAGE, designed to handle arbitrarily heterogeneous and
asynchronous computations. By being robust to "stragglers" and adaptively
ignoring slow computations, Freya PAGE offers significantly improved time
complexity guarantees compared to all previous methods, including Asynchronous
SGD, Rennala SGD, SPIDER, and PAGE, while requiring weaker assumptions. The
algorithm relies on novel generic stochastic gradient collection strategies
with theoretical guarantees that can be of interest on their own, and may be
used in the design of future optimization methods. Furthermore, we establish a
lower bound for smooth nonconvex finite-sum problems in the asynchronous setup,
providing a fundamental time complexity limit. This lower bound is tight and
demonstrates the optimality of Freya PAGE in the large-scale regime, i.e., when
$\sqrt{m} \geq n$, where $n$ is # of workers, and $m$ is # of data samples.

该研究论文介绍了一种名为 Freya PAGE 的新的并行方法，用于处理异构和异步计算情况下的非凸有限和问题，该方法具有较强的鲁棒性，通过忽略慢速计算和对付异常值，相比之前的方法包括异步 SGD、Rennala SGD、SPIDER 和 PAGE，具有显著改善的时间复杂度保证。它还提供了一种新颖的随机梯度收集策略，并建立了异步设置下平滑非凸有限和问题的时间复杂性下界，证明了 Freya PAGE 在大规模情况下的最优性。

Freya PAGE: 大规模非凸有限和优化中的首个最优时间复杂性与异步计算的异构

Freya PAGE: First Optimal Time Complexity for Large-Scale Nonconvex  Finite-Sum Optimization with Heterogeneous Asynchronous Computations

Spiking neural networks are a type of artificial neural networks in which
communication between neurons is only made of events, also called spikes. This
property allows neural networks to make asynchronous and sparse computations
and therefore to drastically decrease energy consumption when run on
specialized hardware. However, training such networks is known to be difficult,
mainly due to the non-differentiability of the spike activation, which prevents
the use of classical backpropagation. This is because state-of-the-art spiking
neural networks are usually derived from biologically-inspired neuron models,
to which are applied machine learning methods for training. Nowadays, research
about spiking neural networks focuses on the design of training algorithms
whose goal is to obtain networks that compete with their non-spiking version on
specific tasks. In this paper, we attempt the symmetrical approach: we modify
the dynamics of a well-known, easily trainable type of recurrent neural network
to make it event-based. This new RNN cell, called the Spiking Recurrent Cell,
therefore communicates using events, i.e. spikes, while being completely
differentiable. Vanilla backpropagation can thus be used to train any network
made of such RNN cell. We show that this new network can achieve performance
comparable to other types of spiking networks in the MNIST benchmark and its
variants, the Fashion-MNIST and the Neuromorphic-MNIST. Moreover, we show that
this new cell makes the training of deep spiking networks achievable.

本文中，我们尝试一种对称方法：我们通过修改一种易于训练的循环神经网络的动力学来使其基于事件。这种新型 RNN 单元，称为 Spiking Recurrent Cell，因此使用事件即 spike 进行通信，同时可完全区分化。因此，可以使用基本反向传播来训练任何由该 RNN 单元组成的网络。我们展示了这种新的网络在 MNIST 基准测试及其变体 Fashion-MNIST 和 Neuromorphic-MNIST 中可以达到与其他类型的脉冲网络可比较的性能，并且这个新单元使得深度脉冲网络的训练成为可能。