Diffusion models are generative models with impressive text-to-image
synthesis capabilities and have spurred a new wave of creative methods for
classical machine learning tasks. However, the best way to harness the
perceptual knowledge of these generative models for visual tasks is still an
open question. Specifically, it is unclear how to use the prompting interface
when applying diffusion backbones to vision tasks. We find that automatically
generated captions can improve text-image alignment and significantly enhance a
model's cross-attention maps, leading to better perceptual performance. Our
approach improves upon the current SOTA in diffusion-based semantic
segmentation on ADE20K and the current overall SOTA in depth estimation on
NYUv2. Furthermore, our method generalizes to the cross-domain setting; we use
model personalization and caption modifications to align our model to the
target domain and find improvements over unaligned baselines. Our object
detection model, trained on Pascal VOC, achieves SOTA results on Watercolor2K.
Our segmentation method, trained on Cityscapes, achieves SOTA results on Dark
Zurich-val and Nighttime Driving.

扩散模型是具有令人印象深刻的文本到图像合成能力的生成模型，对于传统机器学习任务产生了一系列创新方法。然而，如何运用这些生成模型的感知知识来进行视觉任务仍然是一个未解之谜。本研究发现，自动生成的描述能够改善文本图像对齐，并显著提升模型的交叉注意力图，从而提高知觉性能。我们的方法在 ADE20K 数据集上改进了当前最先进的基于扩散的语义分割模型，并在 NYUv2 数据集上改进了当前最先进的深度估计模型。此外，我们的方法适用于跨领域环境；我们使用模型个性化和标题修改来将模型与目标域对齐，并改善了非对齐基准的性能。我们的目标检测模型，在 Pascal VOC 数据集上训练，实现了 Watercolor2K 数据集上的最佳结果。我们的分割方法，在 Cityscapes 数据集上训练，实现了 Dark Zurich-val 和 Nighttime Driving 数据集上的最佳结果。

基于扩散的感知的文本图像对齐

Text-image Alignment for Diffusion-based Perception

Reasoning does not work well when done in isolation from its significance,
both to the needs and interests of an agent and with respect to the wider
world. Moreover, those issues may best be handled with a new sort of data
structure that goes beyond the knowledge base and incorporates aspects of
perceptual knowledge and even more, in which a kind of anticipatory action may
be key.

从代理的需求和利益以及广泛的世界上的重要性来看，孤立无援地进行推理效果不佳。此外，这些问题可能最好通过一种新的数据结构来处理，该数据结构超越了知识库，包含了感知知识的方面，甚至可能是一种预期的行动所关键。