We propose a new approach to non-parametric density estimation, that is based
on regularizing a Sobolev norm of the density. This method is provably
different from Kernel Density Estimation, and makes the bias of the model clear
and interpretable. While there is no closed analytic form for the associated
kernel, we show that one can approximate it using sampling. The optimization
problem needed to determine the density is non-convex, and standard gradient
methods do not perform well. However, we show that with an appropriate
initialization and using natural gradients, one can obtain well performing
solutions. Finally, while the approach provides unnormalized densities, which
prevents the use of log-likelihood for cross validation, we show that one can
instead adapt Fisher Divergence based Score Matching methods for this task. We
evaluate the resulting method on the comprehensive recent Anomaly Detection
benchmark suite, ADBench, and find that it ranks second best, among more than
15 algorithms.

我们提出了一种基于正则化密度的 Sobolev 范数的非参数密度估计的新方法，该方法与核密度估计不同，使模型的偏差清晰且可解释，虽然没有与其关联的核的封闭解析形式，但我们展示了可以使用采样来近似它，所需确定密度的优化问题是非凸的，并且标准梯度法性能不佳，然而我们展示了适当的初始化和使用自然梯度可以得到良好的解，最后，虽然该方法提供未归一化的密度，无法使用对数似然进行交叉验证，但我们展示了可以使用基于费歇分歧的得分匹配方法来完成此任务，我们在最近的安全检测基准套件 ADBenchn 上对所得方法进行了评估，发现排名第二，超过 15 种算法。

隐式归一化显式正则化密度估计

Implicitly Normalized Explicitly Regularized Density Estimation

Efficiently sampling from un-normalized target distributions is a fundamental
problem in scientific computing and machine learning. Traditional approaches
like Markov Chain Monte Carlo (MCMC) guarantee asymptotically unbiased samples
from such distributions but suffer from computational inefficiency,
particularly when dealing with high-dimensional targets, as they require
numerous iterations to generate a batch of samples. In this paper, we propose
an efficient and scalable neural implicit sampler that overcomes these
limitations. Our sampler can generate large batches of samples with low
computational costs by leveraging a neural transformation that directly maps
easily sampled latent vectors to target samples without the need for iterative
procedures. To train the neural implicit sampler, we introduce two novel
methods: the KL training method and the Fisher training method. The former
minimizes the Kullback-Leibler divergence, while the latter minimizes the
Fisher divergence. By employing these training methods, we effectively optimize
the neural implicit sampler to capture the desired target distribution. To
demonstrate the effectiveness, efficiency, and scalability of our proposed
samplers, we evaluate them on three sampling benchmarks with different scales.
These benchmarks include sampling from 2D targets, Bayesian inference, and
sampling from high-dimensional energy-based models (EBMs). Notably, in the
experiment involving high-dimensional EBMs, our sampler produces samples that
are comparable to those generated by MCMC-based methods while being more than
100 times more efficient, showcasing the efficiency of our neural sampler. We
believe that the theoretical and empirical contributions presented in this work
will stimulate further research on developing efficient samplers for various
applications beyond the ones explored in this study.

本文提出了一种高效可扩展的神经隐式采样器，它通过利用神经转换将易于采样的潜在向量直接映射到目标样本，而无需迭代过程，从而实现低计算成本生成大批量样本，此外，本文还引入了 KL 训练方法和 Fisher 训练方法来有效地优化了所提出神经隐式采样器以捕获所需的目标分布。

基于熵的训练方法用于可扩展的神经隐式采样器

Entropy-based Training Methods for Scalable Neural Implicit Sampler

In this paper, we consider an infinite dimensional exponential family,
$\mathcal{P}$ of probability densities, which are parametrized by functions in
a reproducing kernel Hilbert space, $H$ and show it to be quite rich in the
sense that a broad class of densities on $\mathbb{R}^d$ can be approximated
arbitrarily well in Kullback-Leibler (KL) divergence by elements in
$\mathcal{P}$. The main goal of the paper is to estimate an unknown density,
$p_0$ through an element in $\mathcal{P}$. Standard techniques like maximum
likelihood estimation (MLE) or pseudo MLE (based on the method of sieves),
which are based on minimizing the KL divergence between $p_0$ and
$\mathcal{P}$, do not yield practically useful estimators because of their
inability to efficiently handle the log-partition function. Instead, we propose
an estimator, $\hat{p}_n$ based on minimizing the \emph{Fisher divergence},
$J(p_0\Vert p)$ between $p_0$ and $p\in \mathcal{P}$, which involves solving a
simple finite-dimensional linear system. When $p_0\in\mathcal{P}$, we show that
the proposed estimator is consistent, and provide a convergence rate of
$n^{-\min\left\{\frac{2}{3},\frac{2\beta+1}{2\beta+2}\right\}}$ in Fisher
divergence under the smoothness assumption that $\log
p_0\in\mathcal{R}(C^\beta)$ for some $\beta\ge 0$, where $C$ is a certain
Hilbert-Schmidt operator on $H$ and $\mathcal{R}(C^\beta)$ denotes the image of
$C^\beta$. We also investigate the misspecified case of $p_0\notin\mathcal{P}$
and show that $J(p_0\Vert\hat{p}_n)\rightarrow \inf_{p\in\mathcal{P}}J(p_0\Vert
p)$ as $n\rightarrow\infty$, and provide a rate for this convergence under a
similar smoothness condition as above. Through numerical simulations we
demonstrate that the proposed estimator outperforms the non-parametric kernel
density estimator, and that the advantage with the proposed estimator grows as
$d$ increases.

使用无限维度的指数族函数族估算未知的概率密度，通过解决一个简单的有限维度线性系统，获得比非参数核密度估算器更好的结果，该估算器基于 Fisher 散度，可以在 Kullback-Leibler 散度下近似表示广泛类别的概率密度。