Building generalized models that can solve many computer vision tasks
simultaneously is an intriguing direction. Recent works have shown image itself
can be used as a natural interface for general-purpose visual perception and
demonstrated inspiring results. In this paper, we explore diffusion-based
vision generalists, where we unify different types of dense prediction tasks as
conditional image generation and re-purpose pre-trained diffusion models for
it. However, directly applying off-the-shelf latent diffusion models leads to a
quantization issue. Thus, we propose to perform diffusion in pixel space and
provide a recipe for finetuning pre-trained text-to-image diffusion models for
dense vision tasks. In experiments, we evaluate our method on four different
types of tasks and show competitive performance to the other vision
generalists.

构建广义模型以同时解决多个计算机视觉任务是一个有趣的方向，本文探索基于扩散的视觉通用模型，将不同类型的密集预测任务统一为条件图像生成，并重新调整预训练的扩散模型，我们在四种不同类型的任务上实验证明了与其他视觉通用模型竞争性的性能。

面向稠密视觉任务的扩散型通用模型

Toward a Diffusion-Based Generalist for Dense Vision Tasks

Due to the costliness of labelled data in real-world applications,
semi-supervised learning, underpinned by pseudo labelling, is an appealing
solution. However, handling confusing samples is nontrivial: discarding
valuable confusing samples would compromise the model generalisation while
using them for training would exacerbate the issue of confirmation bias caused
by the resulting inevitable mislabelling. To solve this problem, this paper
proposes to use confusing samples proactively without label correction.
Specifically, a Virtual Category (VC) is assigned to each confusing sample in
such a way that it can safely contribute to the model optimisation even without
a concrete label. This provides an upper bound for inter-class information
sharing capacity, which eventually leads to a better embedding space. Extensive
experiments on two mainstream dense prediction tasks -- semantic segmentation
and object detection, demonstrate that the proposed VC learning significantly
surpasses the state-of-the-art, especially when only very few labels are
available. Our intriguing findings highlight the usage of VC learning in dense
vision tasks.

本文提出一种使用虚拟类别 (Virtual Category) 的方法，在处理困惑样本时，无需进行标签修正，从而提高模型的优化能力和嵌入空间的质量，在语义分割和目标检测等密集预测任务中，该方法明显超越了现有技术的状态，尤其在只有很少标签样本的情况下。

虚拟类别学习：一种针对极其有限标签的密集预测的半监督学习方法

Virtual Category Learning: A Semi-Supervised Learning Method for Dense  Prediction with Extremely Limited Labels

We introduce the first multitasking vision transformer adapters that learn
generalizable task affinities which can be applied to novel tasks and domains.
Integrated into an off-the-shelf vision transformer backbone, our adapters can
simultaneously solve multiple dense vision tasks in a parameter-efficient
manner, unlike existing multitasking transformers that are parametrically
expensive. In contrast to concurrent methods, we do not require retraining or
fine-tuning whenever a new task or domain is added. We introduce a task-adapted
attention mechanism within our adapter framework that combines gradient-based
task similarities with attention-based ones. The learned task affinities
generalize to the following settings: zero-shot task transfer, unsupervised
domain adaptation, and generalization without fine-tuning to novel domains. We
demonstrate that our approach outperforms not only the existing convolutional
neural network-based multitasking methods but also the vision transformer-based
ones. Our project page is at https://ivrl.github.io/VTAGML.

我们引入了第一款多任务视觉变换器适配器，它能学习到适用于新任务和领域的通用任务关联性。我们的适配器融合在现成的视觉变换器骨干网络中，能够同时解决多个密集视觉任务，而不像现有的多任务变换器那样需要大量的参数。与同时方法相比，我们无需在添加新任务或领域时重新训练或微调。我们在适配器框架中引入了一种任务适应的注意机制，将基于梯度的任务相似度与基于注意力的相似度相结合。所学习的任务关联性可推广到以下情况：零样本任务迁移、无监督领域适应以及不需要对新领域进行微调的泛化。我们证明了我们的方法不仅优于现有的基于卷积神经网络的多任务方法，还优于基于视觉变换器的方法。