This paper presents a novel method to distill knowledge from a deep pose
regressor network for efficient Visual Odometry (VO). Standard distillation
relies on "dark knowledge" for successful knowledge transfer. As this knowledge
is not available in pose regression and the teacher prediction is not always
accurate, we propose to emphasize the knowledge transfer only when we trust the
teacher. We achieve this by using teacher loss as a confidence score which
places variable relative importance on the teacher prediction. We inject this
confidence score to the main training task via Attentive Imitation Loss (AIL)
and when learning the intermediate representation of the teacher through
Attentive Hint Training (AHT) approach. To the best of our knowledge, this is
the first work which successfully distill the knowledge from a deep pose
regression network. Our evaluation on the KITTI and Malaga dataset shows that
we can keep the student prediction close to the teacher with up to 92.95%
parameter reduction and 2.12x faster in computation time.

本研究提出了一种从深度姿态回归网络中提取知识以实现高效视觉测距的方法，该方法通过 Attentive Imitation Loss (AIL) 的方式注入教师损失作为置信度评分，并使用 Attentive Hint Training (AHT) 方法进行中间表示的学习，以实现知识的迁移。经实验证明，我们可以通过最多 92.95% 的参数减少和 2.12 倍的计算速度来使得学生网络的预测结果保持接近于教师网络。