Vision foundation models are renowned for their generalization ability due to
massive training data. Nevertheless, they demand tremendous training resources,
and the training data is often inaccessible, e.g., CLIP, DINOv2, posing great
challenges to developing derivatives that could advance research in this field.
In this work, we offer a very simple and general solution, named Proteus, to
distill foundation models into smaller equivalents on ImageNet-1K without
access to the original training data. Specifically, we remove the designs from
conventional knowledge distillation settings that result in dataset bias and
present three levels of training objectives, i.e., token, patch, and feature,
to maximize the efficacy of knowledge transfer. In this manner, Proteus is
trained at ImageNet-level costs with surprising ability, facilitating the
accessibility of training foundation models for the broader research community.
Leveraging DINOv2-g/14 as the teacher, Proteus-L/14 matches the performance of
the Oracle method DINOv2-L/14 (142M training data) across 15 benchmarks and
outperforms other vision foundation models including CLIP-L/14 (400M),
OpenCLIP-L/14 (400M/2B) and SynCLR-L/14 (600M).

使用名称为 Proteus 的简单通用解决方案，在无法访问原始训练数据的情况下，通过移除传统知识蒸馏设置中导致数据集偏差的设计，并提供三个级别的训练目标，即令牌、补丁和特征，最大化知识传递的有效性，在 ImageNet 层次的开销下进行训练，从而使广大研究社区能够更轻松地获得训练基础模型的能力。

以 ImageNet 水平成本访问视觉基础模型

Accessing Vision Foundation Models at ImageNet-level Costs

In recent years, portfolio approaches to solving SAT problems and CSPs have
become increasingly common. There are also a number of different encodings for
representing CSPs as SAT instances. In this paper, we leverage advances in both
SAT and CSP solving to present a novel hierarchical portfolio-based approach to
CSP solving, which we call Proteus, that does not rely purely on CSP solvers.
Instead, it may decide that it is best to encode a CSP problem instance into
SAT, selecting an appropriate encoding and a corresponding SAT solver. Our
experimental evaluation used an instance of Proteus that involved four CSP
solvers, three SAT encodings, and six SAT solvers, evaluated on the most
challenging problem instances from the CSP solver competitions, involving
global and intensional constraints. We show that significant performance
improvements can be achieved by Proteus obtained by exploiting alternative
view-points and solvers for combinatorial problem-solving.

该论文提出了一种名为 Proteus 的分层组合策略解决 CSP 问题的方法，该方法能够将 CSP 实例编码为 SAT 形式，并选择合适的编码和相应的 SAT 求解器进行求解，实验结果表明 Proteus 能够显著提高组合问题求解性能。