We propose SING (StabIlized and Normalized Gradient), a plug-and-play
technique that improves the stability and generalization of the Adam(W)
optimizer. SING is straightforward to implement and has minimal computational
overhead, requiring only a layer-wise standardization of the gradients fed to
Adam(W) without introducing additional hyper-parameters. We support the
effectiveness and practicality of the proposed approach by showing improved
results on a wide range of architectures, problems (such as image
classification, depth estimation, and natural language processing), and in
combination with other optimizers. We provide a theoretical analysis of the
convergence of the method, and we show that by virtue of the standardization,
SING can escape local minima narrower than a threshold that is inversely
proportional to the network's depth.

通过对 Adam (W) 进行标准化处理和梯度的标准化，提出了一种插拔式的技术 SING（稳定和标准化梯度），以实现优化器的稳定性和普适性。理论分析和实际实验都支持了该方法的有效性和实用性。

SING: 一种即插即用的深度神经网络学习技术

SING: A Plug-and-Play DNN Learning Technique

Recent progress in deep learning for audio synthesis opens the way to models
that directly produce the waveform, shifting away from the traditional paradigm
of relying on vocoders or MIDI synthesizers for speech or music generation.
Despite their successes, current state-of-the-art neural audio synthesizers
such as WaveNet and SampleRNN suffer from prohibitive training and inference
times because they are based on autoregressive models that generate audio
samples one at a time at a rate of 16kHz. In this work, we study the more
computationally efficient alternative of generating the waveform frame-by-frame
with large strides. We present SING, a lightweight neural audio synthesizer for
the original task of generating musical notes given desired instrument, pitch
and velocity. Our model is trained end-to-end to generate notes from nearly
1000 instruments with a single decoder, thanks to a new loss function that
minimizes the distances between the log spectrograms of the generated and
target waveforms. On the generalization task of synthesizing notes for pairs of
pitch and instrument not seen during training, SING produces audio with
significantly improved perceptual quality compared to a state-of-the-art
autoencoder based on WaveNet as measured by a Mean Opinion Score (MOS), and is
about 32 times faster for training and 2, 500 times faster for inference.

本研究提出了一种基于深度学习的轻量级神经音频合成器 SING，其可以通过单一解码器从近 1000 种乐器中生成音符，并且通过新的损失函数将生成的和目标波形的对数谱距离最小化，提高了音质的感知品质。与基于 WaveNet 的最新自动编码器相比，在训练和推理方面分别快 32 倍和快 2500 倍。