In recent years, cross-modal hashing (CMH) has attracted increasing
attentions, mainly because its potential ability of mapping contents from
different modalities, especially in vision and language, into the same space,
so that it becomes efficient in cross-modal data retrieval. There are two main
frameworks for CMH, differing from each other in whether semantic supervision
is required. Compared to the unsupervised methods, the supervised methods often
enjoy more accurate results, but require much heavier labors in data
annotation. In this paper, we propose a novel approach that enables guiding a
supervised method using outputs produced by an unsupervised method.
Specifically, we make use of teacher-student optimization for propagating
knowledge. Experiments are performed on two popular CMH benchmarks, i.e., the
MIRFlickr and NUS-WIDE datasets. Our approach outperforms all existing
unsupervised methods by a large margin.

本文提出了一种新的方法，通过使用无监督方法产生的输出来指导监督方法，具体来说，使用师生优化来传播知识，在两个流行的 CMH 基准数据集 MIRFlickr 和 NUS-WIDE 上进行了实验，结果表明我们的方法大大优于所有现有的无监督方法。

从无到有：用无监督知识蒸馏实现跨模态哈希

Creating Something from Nothing: Unsupervised Knowledge Distillation for  Cross-Modal Hashing

Optimizing a deep neural network is a fundamental task in computer vision,
yet direct training methods often suffer from over-fitting. Teacher-student
optimization aims at providing complementary cues from a model trained
previously, but these approaches are often considerably slow due to the
pipeline of training a few generations in sequence, i.e., time complexity is
increased by several times.
This paper presents snapshot distillation (SD), the first framework which
enables teacher-student optimization in one generation. The idea of SD is very
simple: instead of borrowing supervision signals from previous generations, we
extract such information from earlier epochs in the same generation, meanwhile
make sure that the difference between teacher and student is sufficiently large
so as to prevent under-fitting. To achieve this goal, we implement SD in a
cyclic learning rate policy, in which the last snapshot of each cycle is used
as the teacher for all iterations in the next cycle, and the teacher signal is
smoothed to provide richer information. In standard image classification
benchmarks such as CIFAR100 and ILSVRC2012, SD achieves consistent accuracy
gain without heavy computational overheads. We also verify that models
pre-trained with SD transfers well to object detection and semantic
segmentation in the PascalVOC dataset.

本研究提出了快照蒸馏 (SD) 框架，它将深度神经网络的教师 - 学生优化方法应用于一个生成周期内，通过循环学习率策略以及利用同一周期内早期时期的信息提取，避免了时间复杂度的增大，同时在 CIFAR100 和 ILSVRC2012 等基准测试中获得了准确性的一致提升。

快照蒸馏：一代中的教师 - 学生优化

Snapshot Distillation: Teacher-Student Optimization in One Generation

We focus on the problem of training a deep neural network in generations. The
flowchart is that, in order to optimize the target network (student), another
network (teacher) with the same architecture is first trained, and used to
provide part of supervision signals in the next stage. While this strategy
leads to a higher accuracy, many aspects (e.g., why teacher-student
optimization helps) still need further explorations.
This paper studies this problem from a perspective of controlling the
strictness in training the teacher network. Existing approaches mostly used a
hard distribution (e.g., one-hot vectors) in training, leading to a strict
teacher which itself has a high accuracy, but we argue that the teacher needs
to be more tolerant, although this often implies a lower accuracy. The
implementation is very easy, with merely an extra loss term added to the
teacher network, facilitating a few secondary classes to emerge and complement
to the primary class. Consequently, the teacher provides a milder supervision
signal (a less peaked distribution), and makes it possible for the student to
learn from inter-class similarity and potentially lower the risk of
over-fitting. Experiments are performed on standard image classification tasks
(CIFAR100 and ILSVRC2012). Although the teacher network behaves less powerful,
the students show a persistent ability growth and eventually achieve higher
classification accuracies than other competitors. Model ensemble and transfer
feature extraction also verify the effectiveness of our approach.

本文从严格性控制角度研究了深度神经网络的教师网络和学生网络的优化问题。通过针对教师网络训练中的宽松性进行优化，使得学生网络在学习过程中更容易获得跨类别相似性信息，从而在 CIFAR100 和 ILSVRC2012 等数据集的图像分类任务中取得更高的分类准确率。