Large language models have evolved data-efficient generalists, benefiting
from the universal language interface and large-scale pre-training. However,
constructing a data-efficient generalist for dense visual prediction presents a
distinct challenge due to the variation in label structures across different
tasks. Consequently, generalization to unseen dense prediction tasks in the
low-data regime is not straightforward and has received less attention from
previous vision generalists. In this study, we explore a universal model that
can flexibly adapt to unseen dense label structures with a few examples,
enabling it to serve as a data-efficient vision generalist in diverse
real-world scenarios. To this end, we base our method on a powerful
meta-learning framework and explore several axes to improve its performance and
versatility for real-world problems, such as flexible adaptation mechanisms and
scalability. We evaluate our model across a spectrum of unseen real-world
scenarios where low-shot learning is desirable, including video, 3D, medical,
biological, and user-interactive tasks. Equipped with a generic architecture
and an effective adaptation mechanism, our model flexibly adapts to all of
these tasks with at most 50 labeled images, showcasing a significant
advancement over existing data-efficient generalist approaches. Codes are
available at this https URL

通过使用元学习框架及一些改进技术，我们构建了一个适应不同稠密标签结构的通用模型，并在各种真实场景中进行了评估，表明我们的模型在最多使用 50 张标记图像的情况下，比现有的数据高效通用方法取得了显著的进展。

变色龙：一种在野外进行密集视觉预测的数据高效通用模型

Chameleon: A Data-Efficient Generalist for Dense Visual Prediction in  the Wild

The emergence of vision transformers (ViTs) in image classification has
shifted the methodologies for visual representation learning. In particular,
ViTs learn visual representation at full receptive field per layer across all
the image patches, in comparison to the increasing receptive fields of CNNs
across layers and other alternatives (e.g., large kernels and atrous
convolution). In this work, for the first time we explore the global context
learning potentials of ViTs for dense visual prediction (e.g., semantic
segmentation). Our motivation is that through learning global context at full
receptive field layer by layer, ViTs may capture stronger long-range dependency
information, critical for dense prediction tasks. We first demonstrate that
encoding an image as a sequence of patches, a vanilla ViT without local
convolution and resolution reduction can yield stronger visual representation
for semantic segmentation. For example, our model, termed as SEgmentation
TRansformer (SETR), excels on ADE20K (50.28% mIoU, the first position in the
test leaderboard on the day of submission) and Pascal Context (55.83% mIoU),
and performs competitively on Cityscapes. For tackling general dense visual
prediction tasks in a cost-effective manner, we further formulate a family of
Hierarchical Local-Global (HLG) Transformers, characterized by local attention
within windows and global-attention across windows in a pyramidal architecture.
Extensive experiments show that our methods achieve appealing performance on a
variety of dense prediction tasks (e.g., object detection and instance
segmentation and semantic segmentation) as well as image classification. Our
code and models are available at this https URL

本研究首次探索视觉变换器（ViT）的全局上下文学习潜力以实现密集视觉预测，编码图像作为一系列补丁并学习全局上下文，使 ViT 可以捕获更强的长距离依赖信息，提出的分割转换器（SETR）在 ADE20K 和 Pascal Context 数据集上取得了出色表现，同时提出了一系列分层本地 - 全局转换器，架构由局部关注（在窗口内）和全局关注（跨窗口）构成，对于各种密集预测任务具有吸引人的性能