This paper investigates an under-explored but important problem: given a
collection of pre-trained neural networks, predicting their performance on each
multi-modal task without fine-tuning them, such as image recognition,
referring, captioning, visual question answering, and text question answering.
A brute-force approach is to finetune all models on all target datasets,
bringing high computational costs. Although recent-advanced approaches employed
lightweight metrics to measure models' transferability,they often depend
heavily on the prior knowledge of a single task, making them inapplicable in a
multi-modal multi-task scenario. To tackle this issue, we propose an efficient
multi-task model selector (EMMS), which employs large-scale foundation models
to transform diverse label formats such as categories, texts, and bounding
boxes of different downstream tasks into a unified noisy label embedding. EMMS
can estimate a model's transferability through a simple weighted linear
regression, which can be efficiently solved by an alternating minimization
algorithm with a convergence guarantee. Extensive experiments on 5 downstream
tasks with 24 datasets show that EMMS is fast, effective, and generic enough to
assess the transferability of pre-trained models, making it the first model
selection method in the multi-task scenario. For instance, compared with the
state-of-the-art method LogME enhanced by our label embeddings, EMMS achieves
9.0\%, 26.3\%, 20.1\%, 54.8\%, 12.2\% performance gain on image recognition,
referring, captioning, visual question answering, and text question answering,
while bringing 5.13x, 6.29x, 3.59x, 6.19x, and 5.66x speedup in wall-clock
time, respectively. The code is available at
this https URL

通过大规模基础模型将不同下游任务的多样化标签格式转换为统一的噪声标签嵌入，从而通过简单的加权线性回归估计模型的可传递性，提出了一个高效的多任务模型选择器 (EMMS)。与现有方法相比，EMMS 在图像识别、引用、字幕生成、视觉问答和文本问答等多个任务中取得了更好的性能，并在耗时方面带来了显著加速。