Deep latent generative models have attracted increasing attention due to the
capacity of combining the strengths of deep learning and probabilistic models
in an elegant way. The data representations learned with the models are often
continuous and dense. However in many applications, sparse representations are
expected, such as learning sparse high dimensional embedding of data in an
unsupervised setting, and learning multi-labels from thousands of candidate
tags in a supervised setting. In some scenarios, there could be further
restriction on degree of sparsity: the number of non-zero features of a
representation cannot be larger than a pre-defined threshold $L_0$. In this
paper we propose a sparse deep latent generative model SDLGM to explicitly
model degree of sparsity and thus enable to learn the sparse structure of the
data with the quantified sparsity constraint. The resulting sparsity of a
representation is not fixed, but fits to the observation itself under the
pre-defined restriction. In particular, we introduce to each observation $i$ an
auxiliary random variable $L_i$, which models the sparsity of its
representation. The sparse representations are then generated with a two-step
sampling process via two Gumbel-Softmax distributions. For inference and
learning, we develop an amortized variational method based on MC gradient
estimator. The resulting sparse representations are differentiable with
backpropagation. The experimental evaluation on multiple datasets for
unsupervised and supervised learning problems shows the benefits of the
proposed method.

本文提出了一种称为 SDLGM 的稀疏深度潜在生成模型，可以通过限制表征的非零特征的数量实现数据的稀疏学习，其生成的稀疏表示可以根据观察本身调整，并使用 Gumbel-Softmax 分布进行两步采样生成，采取蒙特卡罗梯度估计器的变分方法进行推论和学习，实验结果表明其性能优于传统的生成模型。