The goal of interactive image segmentation is to delineate specific regions
within an image via visual or language prompts. Low-latency and high-quality
interactive segmentation with diverse prompts remain challenging for existing
specialist and generalist models. Specialist models, with their limited prompts
and task-specific designs, experience high latency because the image must be
recomputed every time the prompt is updated, due to the joint encoding of image
and visual prompts. Generalist models, exemplified by the Segment Anything
Model (SAM), have recently excelled in prompt diversity and efficiency, lifting
image segmentation to the foundation model era. However, for high-quality
segmentations, SAM still lags behind state-of-the-art specialist models despite
SAM being trained with x100 more segmentation masks. In this work, we delve
deep into the architectural differences between the two types of models. We
observe that dense representation and fusion of visual prompts are the key
design choices contributing to the high segmentation quality of specialist
models. In light of this, we reintroduce this dense design into the generalist
models, to facilitate the development of generalist models with high
segmentation quality. To densely represent diverse visual prompts, we propose
to use a dense map to capture five types: clicks, boxes, polygons, scribbles,
and masks. Thus, we propose SegNext, a next-generation interactive segmentation
approach offering low latency, high quality, and diverse prompt support. Our
method outperforms current state-of-the-art methods on HQSeg-44K and DAVIS,
both quantitatively and qualitatively.

SegNext 是下一代交互式分割方法，具有低延迟、高质量和多样化的支持，通过引入密集设计，细化各种视觉提示的表示，提高了综合型模型的分割质量，在 HQSeg-44K 和 DAVIS 上表现出色。

以低延迟、高质量和多样化提示的方式重新思考交互式图像分割

Rethinking Interactive Image Segmentation with Low Latency, High  Quality, and Diverse Prompts

The potential of large language models (LLMs) to simultaneously perform a
wide range of natural language processing (NLP) tasks has been the subject of
extensive research. Although instruction tuning has proven to be a
data-efficient method for transforming LLMs into such generalist models, their
performance still lags behind specialist models trained exclusively for
specific tasks. In this paper, we investigate whether incorporating
broad-coverage generalist instruction tuning can contribute to building a
specialist model. We hypothesize that its efficacy depends on task specificity
and skill requirements. Our experiments assess four target tasks with distinct
coverage levels, revealing that integrating generalist instruction tuning
consistently enhances model performance when the task coverage is broad. The
effect is particularly pronounced when the amount of task-specific training
data is limited. Further investigation into three target tasks focusing on
different capabilities demonstrates that generalist instruction tuning improves
understanding and reasoning abilities. However, for tasks requiring factual
knowledge, generalist data containing hallucinatory information may negatively
affect the model's performance. Overall, our work provides a systematic guide
for developing specialist models with general instruction tuning. Our code and
other related resources can be found at
this https URL

通过整合广泛覆盖的通用模型调优方式，本研究研究了在建立专用模型方面是否有助于提高性能，结果表明广泛覆盖的任务和有限的任务特定训练数据时，整合通用模型调优可以始终提高模型性能。

专家还是通才？特定 NLP 任务的指导调校

Specialist or Generalist? Instruction Tuning for Specific NLP Tasks

This paper presents a comprehensive survey of the taxonomy and evolution of
multimodal foundation models that demonstrate vision and vision-language
capabilities, focusing on the transition from specialist models to
general-purpose assistants. The research landscape encompasses five core
topics, categorized into two classes. (i) We start with a survey of
well-established research areas: multimodal foundation models pre-trained for
specific purposes, including two topics -- methods of learning vision backbones
for visual understanding and text-to-image generation. (ii) Then, we present
recent advances in exploratory, open research areas: multimodal foundation
models that aim to play the role of general-purpose assistants, including three
topics -- unified vision models inspired by large language models (LLMs),
end-to-end training of multimodal LLMs, and chaining multimodal tools with
LLMs. The target audiences of the paper are researchers, graduate students, and
professionals in computer vision and vision-language multimodal communities who
are eager to learn the basics and recent advances in multimodal foundation
models.

本文综述了多模基础模型的分类和演变，重点关注从专业模型到通用助手的转变。研究领域包括两个核心主题：(i) 针对特定目的预训练的多模基础模型，包括学习视觉骨干进行视觉理解和文本到图像生成的方法；(ii) 旨在充当通用助手的多模基础模型的最新进展，包括受大型语言模型启发的统一视觉模型、多模语言模型的端到端训练以及将多模工具与语言模型相结合的方法。本文面向计算机视觉和视觉语言多模社区的研究人员、研究生和专业人士，他们渴望了解多模基础模型的基础知识和最新进展。