A handful of visual foundation models (VFMs) have recently emerged as the
backbones for numerous downstream tasks. VFMs like CLIP, DINOv2, SAM are
trained with distinct objectives, exhibiting unique characteristics for various
downstream tasks. We find that despite their conceptual differences, these
models can be effectively merged into a unified model through multi-teacher
distillation. We name this approach AM-RADIO (Agglomerative Model -- Reduce All
Domains Into One). This integrative approach not only surpasses the performance
of individual teacher models but also amalgamates their distinctive features,
such as zero-shot vision-language comprehension, detailed pixel-level
understanding, and open vocabulary segmentation capabilities. In pursuit of the
most hardware-efficient backbone, we evaluated numerous architectures in our
multi-teacher distillation pipeline using the same training recipe. This led to
the development of a novel architecture (E-RADIO) that exceeds the performance
of its predecessors and is at least 7x faster than the teacher models. Our
comprehensive benchmarking process covers downstream tasks including ImageNet
classification, ADE20k semantic segmentation, COCO object detection and
LLaVa-1.5 framework.
Code: this https URL

AM-RADIO 是一种通过多教师蒸馏将多个视觉基础模型 (BFM) 有效合并为一个统一模型的方法，超越了单个教师模型的性能，并融合了其独特的特征，如零 - shot 视觉 - 语言理解、像素级理解和开放式词汇分割能力。通过评估多个架构，此方法还开发了一种比以前的模型性能更好且至少快 7 倍的硬件高效背骨 (E-RADIO)。综合基准测试还包括 ImageNet 分类、ADE20k 语义分割、COCO 目标检测和 LLaVa-1.5 框架。