Universal image representations are critical in enabling real-world
fine-grained and instance-level recognition applications, where objects and
entities from any domain must be identified at large scale. Despite recent
advances, existing methods fail to capture important domain-specific knowledge,
while also ignoring differences in data distribution across different domains.
This leads to a large performance gap between efficient universal solutions and
expensive approaches utilising a collection of specialist models, one for each
domain. In this work, we make significant strides towards closing this gap, by
introducing a new learning technique, dubbed UDON (Universal Dynamic Online
DistillatioN). UDON employs multi-teacher distillation, where each teacher is
specialized in one domain, to transfer detailed domain-specific knowledge into
the student universal embedding. UDON's distillation approach is not only
effective, but also very efficient, by sharing most model parameters between
the student and all teachers, where all models are jointly trained in an online
manner. UDON also comprises a sampling technique which adapts the training
process to dynamically allocate batches to domains which are learned slower and
require more frequent processing. This boosts significantly the learning of
complex domains which are characterised by a large number of classes and
long-tail distributions. With comprehensive experiments, we validate each
component of UDON, and showcase significant improvements over the state of the
art in the recent UnED benchmark. Code: this https URL .

UDON 是一种新的学习技术，利用多教师蒸馏将每个教师的特定领域知识传递给学生通用嵌入，通过采样技术动态分配培训批次以加快学习复杂领域的速度，并在最近的 UnED 基准测试中显著提高了性能。