Recent vision foundation models (VFMs) have demonstrated proficiency in
various tasks but require supervised fine-tuning to perform the task of
semantic segmentation effectively. Benchmarking their performance is essential
for selecting current models and guiding future model developments for this
task. The lack of a standardized benchmark complicates comparisons. Therefore,
the primary objective of this paper is to study how VFMs should be benchmarked
for semantic segmentation. To do so, various VFMs are fine-tuned under various
settings, and the impact of individual settings on the performance ranking and
training time is assessed. Based on the results, the recommendation is to
fine-tune the ViT-B variants of VFMs with a 16x16 patch size and a linear
decoder, as these settings are representative of using a larger model, more
advanced decoder and smaller patch size, while reducing training time by more
than 13 times. Using multiple datasets for training and evaluation is also
recommended, as the performance ranking across datasets and domain shifts
varies. Linear probing, a common practice for some VFMs, is not recommended, as
it is not representative of end-to-end fine-tuning. The benchmarking setup
recommended in this paper enables a performance analysis of VFMs for semantic
segmentation. The findings of such an analysis reveal that pretraining with
promptable segmentation is not beneficial, whereas masked image modeling (MIM)
with abstract representations is crucial, even more important than the type of
supervision used. The code for efficiently fine-tuning VFMs for semantic
segmentation can be accessed through the project page at:
this https URL

最近的视觉基础模型在各种任务中展示了高效性，但需要有监督的精调才能有效地执行语义分割任务。本文的主要目标是研究如何对视觉基础模型进行语义分割的基准测试，通过在不同设置下对各种模型进行细调并评估各个设置对性能和训练时间的影响，提出了建议的基准测试设置，以评估视觉基础模型的性能。