The issue of generative pretraining for vision models has persisted as a
long-standing conundrum. At present, the text-to-image (T2I) diffusion model
demonstrates remarkable proficiency in generating high-definition images
matching textual inputs, a feat made possible through its pre-training on
large-scale image-text pairs. This leads to a natural inquiry: can diffusion
models be utilized to tackle visual perception tasks? In this paper, we propose
a simple yet effective scheme to harness a diffusion model for visual
perception tasks. Our key insight is to introduce learnable embeddings (meta
prompts) to the pre-trained diffusion models to extract proper features for
perception. The effect of meta prompts are two-fold. First, as a direct
replacement of the text embeddings in the T2I models, it can activate
task-relevant features during feature extraction. Second, it will be used to
re-arrange the extracted features to ensures that the model focuses on the most
pertinent features for the task on hand. Additionally, we design a recurrent
refinement training strategy that fully leverages the property of diffusion
models, thereby yielding stronger visual features. Extensive experiments across
various benchmarks validate the effectiveness of our approach. Our approach
achieves new performance records in depth estimation tasks on NYU depth V2 and
KITTI, and in semantic segmentation task on CityScapes. Concurrently, the
proposed method attains results comparable to the current state-of-the-art in
semantic segmentation on ADE20K and pose estimation on COCO datasets, further
exemplifying its robustness and versatility.

通过引入可学习的嵌入（元提示）来利用扩散模型解决视觉感知任务，我们的方法在深度估计和语义分割任务上取得了新的性能记录，并在 ADE20K 的语义分割和 COCO 数据集的姿态估计等方面达到了与最先进方法相媲美的结果，展示了其稳健性和多功能性。

利用扩散模型和元提示进行视觉感知

Harnessing Diffusion Models for Visual Perception with Meta Prompts

Neural Radiance Field (NeRF) is a popular method in representing 3D scenes by
optimising a continuous volumetric scene function. Its large success which lies
in applying volumetric rendering (VR) is also its Achilles' heel in producing
view-dependent effects. As a consequence, glossy and transparent surfaces often
appear murky. A remedy to reduce these artefacts is to constrain this VR
equation by excluding volumes with back-facing normal. While this approach has
some success in rendering glossy surfaces, translucent objects are still poorly
represented. In this paper, we present an alternative to the physics-based VR
approach by introducing a self-attention-based framework on volumes along a
ray. In addition, inspired by modern game engines which utilise Light Probes to
store local lighting passing through the scene, we incorporate Learnable
Embeddings to capture view dependent effects within the scene. Our method,
which we call ABLE-NeRF, significantly reduces `blurry' glossy surfaces in
rendering and produces realistic translucent surfaces which lack in prior art.
In the Blender dataset, ABLE-NeRF achieves SOTA results and surpasses Ref-NeRF
in all 3 image quality metrics PSNR, SSIM, LPIPS.

本文提出了一种新的基于自我注意力和可学习嵌入的 3D 场景渲染方法，称为 ABLE-NeRF，以改善之前 NeRF 方法在光泽和半透明表面表现较差的问题，并在渲染结果上获得了比之前先进的结果。