In this work we show that Evolution Strategies (ES) are a viable method for
learning non-differentiable parameters of large supervised models. ES are
black-box optimization algorithms that estimate distributions of model
parameters; however they have only been used for relatively small problems so
far. We show that it is possible to scale ES to more complex tasks and models
with millions of parameters. While using ES for differentiable parameters is
computationally impractical (although possible), we show that a hybrid approach
is practically feasible in the case where the model has both differentiable and
non-differentiable parameters. In this approach we use standard gradient-based
methods for learning differentiable weights, while using ES for learning
non-differentiable parameters - in our case sparsity masks of the weights. This
proposed method is surprisingly competitive, and when parallelized over
multiple devices has only negligible training time overhead compared to
training with gradient descent. Additionally, this method allows to train
sparse models from the first training step, so they can be much larger than
when using methods that require training dense models first. We present results
and analysis of supervised feed-forward models (such as MNIST and CIFAR-10
classification), as well as recurrent models, such as SparseWaveRNN for
text-to-speech.

该研究论文展示了演化策略在学习大型监督模型的非可微参数方面的优越性，尤其是当模型具有百万维参数时，该方法的竞争能力非常出色。这种方法允许瘦模型从第一步开始就可以训练，非常适合于大算力场景。