StyleGAN family is one of the most popular Generative Adversarial Networks
(GANs) for unconditional generation. Despite its impressive performance, its
high demand on storage and computation impedes their deployment on
resource-constrained devices. This paper provides a comprehensive study of
distilling from the popular StyleGAN-like architecture. Our key insight is that
the main challenge of StyleGAN distillation lies in the output discrepancy
issue, where the teacher and student model yield different outputs given the
same input latent code. Standard knowledge distillation losses typically fail
under this heterogeneous distillation scenario. We conduct thorough analysis
about the reasons and effects of this discrepancy issue, and identify that the
mapping network plays a vital role in determining semantic information of
generated images. Based on this finding, we propose a novel initialization
strategy for the student model, which can ensure the output consistency to the
maximum extent. To further enhance the semantic consistency between the teacher
and student model, we present a latent-direction-based distillation loss that
preserves the semantic relations in latent space. Extensive experiments
demonstrate the effectiveness of our approach in distilling StyleGAN2 and
StyleGAN3, outperforming existing GAN distillation methods by a large margin.

这篇论文提出了一种基于输出一致性和潜在空间的语义关系的潜在方向的蒸馏方法，以解决 StyleGAN 的蒸馏问题，该方法在蒸馏 StyleGAN2 和 StyleGAN3 方面的效果显著优于现有的 GAN 蒸馏方法。

在精简 StyleGAN 时要注意差距

Mind the Gap in Distilling StyleGANs

We present a new approach to 3D object representation where a neural network
encodes the geometry of an object directly into the weights and biases of a
second 'mapping' network. This mapping network can be used to reconstruct an
object by applying its encoded transformation to points randomly sampled from a
simple geometric space, such as the unit sphere. We study the effectiveness of
our method through various experiments on subsets of the ShapeNet dataset. We
find that the proposed approach can reconstruct encoded objects with accuracy
equal to or exceeding state-of-the-art methods with orders of magnitude fewer
parameters. Our smallest mapping network has only about 7000 parameters and
shows reconstruction quality on par with state-of-the-art object decoder
architectures with millions of parameters. Further experiments on feature
mixing through the composition of learned functions show that the encoding
captures a meaningful subspace of objects.

通过神经网络将几何结构编码到权重和偏置中，通过映射网络重构对象，可在 7000 参数的情况下达到与百万参数状态下的最新型物体解码器架构同等甚至更高的精度。