We present FIND, a generalized interface for aligning foundation models'
embeddings. As shown in teaser figure, a lightweight transformer interface
without tuning any foundation model weights is enough for a unified image
(segmentation) and dataset-level (retrieval) understanding. The proposed
interface has the following favorable attributes: (1) Generalizable. It applies
to various tasks spanning retrieval, segmentation, \textit{etc.}, under the
same architecture and weights. (2) Prototypable. Different tasks are able to be
implemented through prototyping attention masks and embedding types. (3)
Extendable. The proposed interface is adaptive to new tasks, and new models.
(4) Interleavable. With the benefit of multi-task multi-modal training, the
proposed interface creates an interleaved shared embedding space. In light of
the interleaved embedding space, we introduce the FIND-Bench, which introduces
new training and evaluation annotations to the COCO dataset for interleave
segmentation and retrieval. Our approach achieves state-of-the-art performance
on FIND-Bench and competitive performance on standard retrieval and
segmentation settings. The training, evaluation, and demo code as well as the
dataset have been released at this https URL

我们介绍了 FIND，一个用于对齐基础模型嵌入的广义接口。该接口具有通用性，适用于涵盖检索、分割等各种任务，在相同的架构和权重下。通过原型化注意掩码和嵌入类型，不同的任务可以实施。该接口对新任务和新模型具有适应性。通过多任务多模态训练，该接口创建了一个交替共享的嵌入空间。在 FIND-Bench 上，我们的方法达到了最先进的性能，并在标准的检索和分割设置上取得了有竞争力的性能。

基础模型嵌入的接口

Interfacing Foundation Models' Embeddings

Since obtaining a perfect training dataset (i.e., a dataset which is
considerably large, unbiased, and well-representative of unseen cases) is
hardly possible, many real-world text classifiers are trained on the available,
yet imperfect, datasets. These classifiers are thus likely to have undesirable
properties. For instance, they may have biases against some sub-populations or
may not work effectively in the wild due to overfitting. In this paper, we
propose FIND -- a framework which enables humans to debug deep learning text
classifiers by disabling irrelevant hidden features. Experiments show that by
using FIND, humans can improve CNN text classifiers which were trained under
different types of imperfect datasets (including datasets with biases and
datasets with dissimilar train-test distributions).

提出了一个名为 FIND 的框架，使其能通过禁用无关的隐藏特征来调试在不同类型的不完美数据集下训练的 CNN 文本分类器，从而改进文本分类器。