Deep neural networks (DNNs) provide useful models of visual representational
transformations. We present a method that enables a DNN (student) to learn from
the internal representational spaces of a reference model (teacher), which
could be another DNN or, in the future, a biological brain. Representational
spaces of the student and the teacher are characterized by representational
distance matrices (RDMs). We propose representational distance learning (RDL),
a stochastic gradient descent method that drives the RDMs of the student to
approximate the RDMs of the teacher. We demonstrate that RDL is competitive
with other transfer learning techniques for two publicly available benchmark
computer vision datasets (MNIST and CIFAR-100), while allowing for
architectural differences between student and teacher. By pulling the student's
RDMs towards those of the teacher, RDL significantly improved visual
classification performance when compared to baseline networks that did not use
transfer learning. In the future, RDL may enable combined supervised training
of deep neural networks using task constraints (e.g. images and category
labels) and constraints from brain-activity measurements, so as to build models
that replicate the internal representational spaces of biological brains.

该研究提出了一种称为 “代表空间距离学习” 的方法，该方法可用于将一种深度神经网络模型学习到的内部表现空间应用到另一种模型之中，通过该方法可将学生模型的表现空间序列逐步拟合为更接近于教师模型表现空间的序列。通过将学生模型的表现空间逐渐接近于教师模型的表现空间，使用该方法可以显著提高可视化分类性能。未来该方法可以应用于将深度神经网络的任务约束和生物大脑的约束相结合，从而构建出可以复制生物大脑内部表现空间的深度神经网络模型。