Variational inference with normalizing flows (NFs) is an increasingly popular
alternative to MCMC methods. In particular, NFs based on coupling layers (Real
NVPs) are frequently used due to their good empirical performance. In theory,
increasing the depth of normalizing flows should lead to more accurate
posterior approximations. However, in practice, training deep normalizing flows
for approximating high-dimensional posterior distributions is often infeasible
due to the high variance of the stochastic gradients. In this work, we show
that previous methods for stabilizing the variance of stochastic gradient
descent can be insufficient to achieve stable training of Real NVPs. As the
source of the problem, we identify that, during training, samples often exhibit
unusual high values. As a remedy, we propose a combination of two methods: (1)
soft-thresholding of the scale in Real NVPs, and (2) a bijective soft log
transformation of the samples. We evaluate these and other previously proposed
modification on several challenging target distributions, including a
high-dimensional horseshoe logistic regression model. Our experiments show that
with our modifications, stable training of Real NVPs for posteriors with
several thousand dimensions is possible, allowing for more accurate marginal
likelihood estimation via importance sampling. Moreover, we evaluate several
common training techniques and architecture choices and provide practical
advise for training NFs for high-dimensional variational inference.

通过使用软阈值和双射软对数变换方法，本文提出了稳定训练 Real NVPs 并在高维情景下实现更准确的后验近似的方法。

高维变分推理的稳定正则流训练

Stable Training of Normalizing Flows for High-dimensional Variational  Inference

Density estimation based anomaly detection schemes typically model anomalies
as examples that reside in low-density regions. We propose a modified density
estimation problem and demonstrate its effectiveness for anomaly detection.
Specifically, we assume the density function of normal samples is uniform in
some compact domain. This assumption implies the density function is more
stable (with lower variance) around normal samples than anomalies. We first
corroborate this assumption empirically using a wide range of real-world data.
Then, we design a variance stabilized density estimation problem for maximizing
the likelihood of the observed samples while minimizing the variance of the
density around normal samples. We introduce an ensemble of autoregressive
models to learn the variance stabilized distribution. Finally, we perform an
extensive benchmark with 52 datasets demonstrating that our method leads to
state-of-the-art results while alleviating the need for data-specific
hyperparameter tuning.

通过将正常样本的密度函数假设为在某些紧凑区域内是均匀的，减小方差放宽了对于特定数据的超参数调节要求，提出了基于改进密度估计的异常检测方案，并设计了一个方差稳定的密度估计问题，然后使用自回归模型学习了方差稳定分布，最后在 52 个数据集的广泛基准测试中表现出了最新的结果。