Transformer-based architectures have become the de-facto standard models for
diverse vision tasks owing to their superior performance. As the size of the
models continues to scale up, model distillation becomes extremely important in
various real applications, particularly on devices limited by computational
resources. However, prevailing knowledge distillation methods exhibit
diminished efficacy when confronted with a large capacity gap between the
teacher and the student, e.g, 10x compression rate. In this paper, we present a
novel approach named Automatic Multi-step Distillation (AMD) for large-scale
vision model compression. In particular, our distillation process unfolds
across multiple steps. Initially, the teacher undergoes distillation to form an
intermediate teacher-assistant model, which is subsequently distilled further
to the student. An efficient and effective optimization framework is introduced
to automatically identify the optimal teacher-assistant that leads to the
maximal student performance. We conduct extensive experiments on multiple image
classification datasets, including CIFAR-10, CIFAR-100, and ImageNet. The
findings consistently reveal that our approach outperforms several established
baselines, paving a path for future knowledge distillation methods on
large-scale vision models.

我们提出了一种名为 AMD 的新方法，用于大规模视觉模型压缩，通过跨多个步骤展开蒸馏过程，形成中间的助教模型，以进一步蒸馏到学生模型。我们的方法在多个图像分类数据集上表现优于已有方法，为大规模视觉模型的知识蒸馏方法铺平了道路。

AMD：大规模视觉模型的自动多步蒸馏

AMD: Automatic Multi-step Distillation of Large-scale Vision Models

Multi-domain fake news detection aims to identify whether various news from
different domains is real or fake and has become urgent and important. However,
existing methods are dedicated to improving the overall performance of fake
news detection, ignoring the fact that unbalanced data leads to disparate
treatment for different domains, i.e., the domain bias problem. To solve this
problem, we propose the Dual-Teacher De-biasing Distillation framework (DTDBD)
to mitigate bias across different domains. Following the knowledge distillation
methods, DTDBD adopts a teacher-student structure, where pre-trained large
teachers instruct a student model. In particular, the DTDBD consists of an
unbiased teacher and a clean teacher that jointly guide the student model in
mitigating domain bias and maintaining performance. For the unbiased teacher,
we introduce an adversarial de-biasing distillation loss to instruct the
student model in learning unbiased domain knowledge. For the clean teacher, we
design domain knowledge distillation loss, which effectively incentivizes the
student model to focus on representing domain features while maintaining
performance. Moreover, we present a momentum-based dynamic adjustment algorithm
to trade off the effects of two teachers. Extensive experiments on Chinese and
English datasets show that the proposed method substantially outperforms the
state-of-the-art baseline methods in terms of bias metrics while guaranteeing
competitive performance.

提出了一种基于 Dual-Teacher De-biasing Distillation 框架的多领域假新闻检测方法，通过知识蒸馏方法构建了一个不带偏见的教师和一个具有领域知识的干净教师来指导学生模型，以减少领域偏见问题并保持性能，实验证明该方法在偏见度量上明显优于现有基准方法，并保证了具有竞争性的性能。

多领域假新闻检测的双教师去偏执蒸馏框架

Dual-Teacher De-biasing Distillation Framework for Multi-domain Fake  News Detection

For semantic segmentation, most existing real-time deep models trained with
each frame independently may produce inconsistent results for a video sequence.
Advanced methods take into considerations the correlations in the video
sequence, e.g., by propagating the results to the neighboring frames using
optical flow, or extracting the frame representations with other frames, which
may lead to inaccurate results or unbalanced latency. In this work, we process
efficient semantic video segmentation in a per-frame fashion during the
inference process. Different from previous per-frame models, we explicitly
consider the temporal consistency among frames as extra constraints during the
training process and embed the temporal consistency into the segmentation
network. Therefore, in the inference process, we can process each frame
independently with no latency, and improve the temporal consistency with no
extra computational cost and post-processing. We employ compact models for
real-time execution. To narrow the performance gap between compact models and
large models, new knowledge distillation methods are designed. Our results
outperform previous keyframe based methods with a better trade-off between the
accuracy and the inference speed on popular benchmarks, including the
Cityscapes and Camvid. The temporal consistency is also improved compared with
corresponding baselines which are trained with each frame independently. Code
is available at: this https URL

本文提出了一种有效的语义视频分割方法，借鉴于现有方法中将结果传播到相邻帧或使用其他帧提取帧表示时可能会导致不准确的结果或不平衡的延迟问题，我们在推理过程中采用逐帧方式处理，其中显式考虑帧与帧之间的时间一致性，并将此一致性嵌入到语义分割网络中。通过新的知识蒸馏方法，我们成功地缩小了紧凑型模型与大型模型之间的性能差距。我们的结果在 Cityscapes 和 Camvid 等流行基准测试中优于先前的基于关键帧的方法，并提高了与独立训练每帧的对应基线的时间一致性。