Deep generative models (DGMs) of images are now sufficiently mature that they
produce nearly photorealistic samples and obtain scores similar to the data
distribution on heuristics such as Frechet Inception Distance (FID). These
results, especially on large-scale datasets such as ImageNet, suggest that DGMs
are learning the data distribution in a perceptually meaningful space and can
be used in downstream tasks. To test this latter hypothesis, we use
class-conditional generative models from a number of model
classes---variational autoencoders, autoregressive models, and generative
adversarial networks (GANs)---to infer the class labels of real data. We
perform this inference by training an image classifier using only synthetic
data and using the classifier to predict labels on real data. The performance
on this task, which we call Classification Accuracy Score (CAS), reveals some
surprising results not identified by traditional metrics and constitute our
contributions. First, when using a state-of-the-art GAN (BigGAN-deep), Top-1
and Top-5 accuracy decrease by 27.9\% and 41.6\%, respectively, compared to the
original data; and conditional generative models from other model classes, such
as Vector-Quantized Variational Autoencoder-2 (VQ-VAE-2) and Hierarchical
Autoregressive Models (HAMs), substantially outperform GANs on this benchmark.
Second, CAS automatically surfaces particular classes for which generative
models failed to capture the data distribution, and were previously unknown in
the literature. Third, we find traditional GAN metrics such as Inception Score
(IS) and FID neither predictive of CAS nor useful when evaluating non-GAN
models. Furthermore, in order to facilitate better diagnoses of generative
models, we open-source the proposed metric.

本研究使用多种类别的条件生成模型推断真实数据的类别标签，并使用分类器在真实数据上进行标签预测，提出了分类准确度评分（CAS）作为新的模型评价指标。实验证明基于自注意力模型的生成模型在 CAS 上显著优于反卷积生成对抗网络模型。