Efficient audio synthesis is an inherently difficult machine learning task,
as human perception is sensitive to both global structure and fine-scale
waveform coherence. Autoregressive models, such as WaveNet, model local
structure at the expense of global latent structure and slow iterative
sampling, while Generative Adversarial Networks (GANs), have global latent
conditioning and efficient parallel sampling, but struggle to generate
locally-coherent audio waveforms. Herein, we demonstrate that GANs can in fact
generate high-fidelity and locally-coherent audio by modeling log magnitudes
and instantaneous frequencies with sufficient frequency resolution in the
spectral domain. Through extensive empirical investigations on the NSynth
dataset, we demonstrate that GANs are able to outperform strong WaveNet
baselines on automated and human evaluation metrics, and efficiently generate
audio several orders of magnitude faster than their autoregressive
counterparts.

本文介绍了在频谱域中建模日志幅度和瞬时频率可提高 GAN 在音频合成方面的表现，并通过 NSynth 数据集的大量实证研究表明，GAN 能够比强 WaveNet 基线在自动化和人工评估指标上表现更好，而且比自回归模型产生的音频快几个数量级。

GANSynth：对抗神经音频合成

GANSynth: Adversarial Neural Audio Synthesis

Generative models in vision have seen rapid progress due to algorithmic
improvements and the availability of high-quality image datasets. In this
paper, we offer contributions in both these areas to enable similar progress in
audio modeling. First, we detail a powerful new WaveNet-style autoencoder model
that conditions an autoregressive decoder on temporal codes learned from the
raw audio waveform. Second, we introduce NSynth, a large-scale and high-quality
dataset of musical notes that is an order of magnitude larger than comparable
public datasets. Using NSynth, we demonstrate improved qualitative and
quantitative performance of the WaveNet autoencoder over a well-tuned spectral
autoencoder baseline. Finally, we show that the model learns a manifold of
embeddings that allows for morphing between instruments, meaningfully
interpolating in timbre to create new types of sounds that are realistic and
expressive.

本研究提出了一种基于 WaveNet 风格自编码器的新型音频模型，进一步介绍了一个规模较大质量较高的音乐数据集 NSynth。使用 NSynth 数据集，我们证明 WaveNet 自编码器比使用频谱自编码器基线的性能有显著提高，同时演示了该模型学习嵌入的能力，实现了在音色上的差值并创造了新类型的逼真和有表现力的声音。