Training gaze following models requires a large number of images with gaze
target coordinates annotated by human annotators, which is a laborious and
inherently ambiguous process. We propose the first semi-supervised method for
gaze following by introducing two novel priors to the task. We obtain the first
prior using a large pretrained Visual Question Answering (VQA) model, where we
compute Grad-CAM heatmaps by `prompting' the VQA model with a gaze following
question. These heatmaps can be noisy and not suited for use in training. The
need to refine these noisy annotations leads us to incorporate a second prior.
We utilize a diffusion model trained on limited human annotations and modify
the reverse sampling process to refine the Grad-CAM heatmaps. By tuning the
diffusion process we achieve a trade-off between the human annotation prior and
the VQA heatmap prior, which retains the useful VQA prior information while
exhibiting similar properties to the training data distribution. Our method
outperforms simple pseudo-annotation generation baselines on the GazeFollow
image dataset. More importantly, our pseudo-annotation strategy, applied to a
widely used supervised gaze following model (VAT), reduces the annotation need
by 50%. Our method also performs the best on the VideoAttentionTarget dataset.

通过引入两个新的先验条件，我们提出了首个半监督的凝视追踪方法。我们利用大型预训练的视觉问答模型计算 Grad-CAM 热图，并通过修正扩散模型中的反向采样过程来改进热图。我们的方法在 GazeFollow 图像数据集上胜过简单的伪标注生成基线，并将注释需求降低了 50%。在 VideoAttentionTarget 数据集上，我们的方法也取得了最好的表现。

扩散优化的半监督凝视跟踪 VQA 注释

Diffusion-Refined VQA Annotations for Semi-Supervised Gaze Following

The popularity of Contrastive Language-Image Pre-training (CLIP) has
propelled its application to diverse downstream vision tasks. To improve its
capacity on downstream tasks, few-shot learning has become a widely-adopted
technique. However, existing methods either exhibit limited performance or
suffer from excessive learnable parameters. In this paper, we propose APE, an
Adaptive Prior rEfinement method for CLIP's pre-trained knowledge, which
achieves superior accuracy with high computational efficiency. Via a prior
refinement module, we analyze the inter-class disparity in the downstream data
and decouple the domain-specific knowledge from the CLIP-extracted cache model.
On top of that, we introduce two model variants, a training-free APE and a
training-required APE-T. We explore the trilateral affinities between the test
image, prior cache model, and textual representations, and only enable a
lightweight category-residual module to be trained. For the average accuracy
over 11 benchmarks, both APE and APE-T attain state-of-the-art and respectively
outperform the second-best by +1.59% and +1.99% under 16 shots with x30 less
learnable parameters.

本文提出了一种自适应先验精细化方法 APE，通过先验精细化模块分析下游数据中的类间差异，将领域特定知识与 CLIP 提取的缓存模型分离，进而在轻量化的类别残差模块的支持下，探究测试图像、先前缓存模型和文本表示之间的三元亲和性，并通过 APE 和 APE-T 这两种模型实现了在 11 个基准数据集上超过第二名 +1.59% 和 +1.99% 的最优性能，且它们使用的学习参数比第二名少 30 倍。