Even nowadays, where Deep Learning (DL) has achieved state-of-the-art
performance in a wide range of research domains, accelerating training and
building robust DL models remains a challenging task. To this end, generations
of researchers have pursued to develop robust methods for training DL
architectures that can be less sensitive to weight distributions, model
architectures and loss landscapes. However, such methods are limited to
adaptive learning rate optimizers, initialization schemes, and clipping
gradients without investigating the fundamental rule of parameters update.
Although multiplicative updates have contributed significantly to the early
development of machine learning and hold strong theoretical claims, to best of
our knowledge, this is the first work that investigate them in context of DL
training acceleration and robustness. In this work, we propose an optimization
framework that fits to a wide range of optimization algorithms and enables one
to apply alternative update rules. To this end, we propose a novel
multiplicative update rule and we extend their capabilities by combining it
with a traditional additive update term, under a novel hybrid update method. We
claim that the proposed framework accelerates training, while leading to more
robust models in contrast to traditionally used additive update rule and we
experimentally demonstrate their effectiveness in a wide range of task and
optimization methods. Such tasks ranging from convex and non-convex
optimization to difficult image classification benchmarks applying a wide range
of traditionally used optimization methods and Deep Neural Network (DNN)
architectures.

本文研究了深度学习训练加速和鲁棒性问题，提出了一种优化框架，其中使用了一种新的乘法更新规则，并将其与传统的加性更新项相结合，实验表明该框架可以应用于广泛的优化方法和深度神经网络结构，并加速训练同时提高了模型的鲁棒性。

加速深度学习训练和提高鲁棒性的乘法更新规则

Multiplicative update rules for accelerating deep learning training and  increasing robustness

This work studies external regret in sequential prediction games with both
positive and negative payoffs. External regret measures the difference between
the payoff obtained by the forecasting strategy and the payoff of the best
action. In this setting, we derive new and sharper regret bounds for the
well-known exponentially weighted average forecaster and for a new forecaster
with a different multiplicative update rule. Our analysis has two main
advantages: first, no preliminary knowledge about the payoff sequence is
needed, not even its range; second, our bounds are expressed in terms of sums
of squared payoffs, replacing larger first-order quantities appearing in
previous bounds. In addition, our most refined bounds have the natural and
desirable property of being stable under rescalings and general translations of
the payoff sequence.

该研究探讨了带有正负回报的顺序预测游戏中外部遗憾的问题，应用了加权平均预测和不同乘法更新规则的新预测方法，并得到了更精确的遗憾界限。