Knowledge distillation is commonly employed to compress neural networks,
reducing the inference costs and memory footprint. In the scenario of
homogenous architecture, feature-based methods have been widely validated for
their effectiveness. However, in scenarios where the teacher and student models
are of heterogeneous architectures, the inherent differences in feature
representation significantly degrade the performance of these methods. Recent
studies have highlighted that low-frequency components constitute the majority
of image features. Motivated by this, we propose a Low-Frequency
Components-based Contrastive Knowledge Distillation (LFCC) framework that
significantly enhances the performance of feature-based distillation between
heterogeneous architectures. Specifically, we designe a set of multi-scale
low-pass filters to extract the low-frequency components of intermediate
features from both the teacher and student models, aligning them in a compact
space to overcome architectural disparities. Moreover, leveraging the intrinsic
pairing characteristic of the teacher-student framework, we design an
innovative sample-level contrastive learning framework that adeptly
restructures the constraints of within-sample feature similarity and
between-sample feature divergence into a contrastive learning task. This
strategy enables the student model to capitalize on intra-sample feature
congruence while simultaneously enhancing the discrimination of features among
disparate samples. Consequently, our LFCC framework accurately captures the
commonalities in feature representation across heterogeneous architectures.
Extensive evaluations and empirical analyses across three architectures (CNNs,
Transformers, and MLPs) demonstrate that LFCC achieves superior performance on
the challenging benchmarks of ImageNet-1K and CIFAR-100. All codes will be
publicly available.

基于低频部分的对比知识蒸馏框架能够更好地在异构架构下提取特征表示的共性。通过使用多尺度低通滤波器提取教师和学生模型中间特征的低频部分，并通过对比学习任务优化学生模型的特征区分度，该框架在 ImageNet-1K 和 CIFAR-100 数据集上展现出卓越的性能。

在紧凑空间中对齐：异构架构之间的对比知识蒸馏

Aligning in a Compact Space: Contrastive Knowledge Distillation between  Heterogeneous Architectures

This paper enables high-fidelity, transferable NeRF editing by frequency
decomposition. Recent NeRF editing pipelines lift 2D stylization results to 3D
scenes while suffering from blurry results, and fail to capture detailed
structures caused by the inconsistency between 2D editings. Our critical
insight is that low-frequency components of images are more
multiview-consistent after editing compared with their high-frequency parts.
Moreover, the appearance style is mainly exhibited on the low-frequency
components, and the content details especially reside in high-frequency parts.
This motivates us to perform editing on low-frequency components, which results
in high-fidelity edited scenes. In addition, the editing is performed in the
low-frequency feature space, enabling stable intensity control and novel scene
transfer. Comprehensive experiments conducted on photorealistic datasets
demonstrate the superior performance of high-fidelity and transferable NeRF
editing. The project page is at https://aigc3d.github.io/freditor.

通过频率分解实现高保真、可迁移的 NeRF 编辑。