Density estimation, a central problem in machine learning, can be performed
using Normalizing Flows (NFs). NFs comprise a sequence of invertible
transformations, that turn a complex target distribution into a simple one, by
exploiting the change of variables theorem. Neural Autoregressive Flows (NAFs)
and Block Neural Autoregressive Flows (B-NAFs) are arguably the most perfomant
members of the NF family. However, they suffer scalability issues and training
instability due to the constraints imposed on the network structure. In this
paper, we propose a novel solution to these challenges by exploiting
transformers to define a new class of neural flows called Transformer Neural
Autoregressive Flows (T-NAFs). T-NAFs treat each dimension of a random variable
as a separate input token, using attention masking to enforce an autoregressive
constraint. We take an amortization-inspired approach where the transformer
outputs the parameters of an invertible transformation. The experimental
results demonstrate that T-NAFs consistently match or outperform NAFs and
B-NAFs across multiple datasets from the UCI benchmark. Remarkably, T-NAFs
achieve these results using an order of magnitude fewer parameters than
previous approaches, without composing multiple flows.

使用 Transformer 神经自回归流（T-NAFs）通过注意力掩码将每个随机变量的维度视为单独的输入令牌，取得了 UCI 基准数据集上与 NAFs 和 B-NAFs 相媲美甚至更好的实验结果，同时使用数量级较少的参数而无需合成多个流。

Transformer 神经自回归流

Transformer Neural Autoregressive Flows

Normalizing flows and autoregressive models have been successfully combined
to produce state-of-the-art results in density estimation, via Masked
Autoregressive Flows (MAF), and to accelerate state-of-the-art WaveNet-based
speech synthesis to 20x faster than real-time, via Inverse Autoregressive Flows
(IAF). We unify and generalize these approaches, replacing the (conditionally)
affine univariate transformations of MAF/IAF with a more general class of
invertible univariate transformations expressed as monotonic neural networks.
We demonstrate that the proposed neural autoregressive flows (NAF) are
universal approximators for continuous probability distributions, and their
greater expressivity allows them to better capture multimodal target
distributions. Experimentally, NAF yields state-of-the-art performance on a
suite of density estimation tasks and outperforms IAF in variational
autoencoders trained on binarized MNIST.

本篇研究提出了使用单调神经网络替代对条件仿射单变量转换的方法进行流式变换，称之为神经自回归流（NAF）。实验结果显示，NAF 在密度估计任务上表现出色，并在 MNIST 二值化数据集上的变分自编码器训练中表现出优于 IAF 的性能。