Training gaze following models requires a large number of images with gaze
target coordinates annotated by human annotators, which is a laborious and
inherently ambiguous process. We propose the first semi-supervised method for
gaze following by introducing two novel priors to the task. We obtain the first
prior using a large pretrained Visual Question Answering (VQA) model, where we
compute Grad-CAM heatmaps by `prompting' the VQA model with a gaze following
question. These heatmaps can be noisy and not suited for use in training. The
need to refine these noisy annotations leads us to incorporate a second prior.
We utilize a diffusion model trained on limited human annotations and modify
the reverse sampling process to refine the Grad-CAM heatmaps. By tuning the
diffusion process we achieve a trade-off between the human annotation prior and
the VQA heatmap prior, which retains the useful VQA prior information while
exhibiting similar properties to the training data distribution. Our method
outperforms simple pseudo-annotation generation baselines on the GazeFollow
image dataset. More importantly, our pseudo-annotation strategy, applied to a
widely used supervised gaze following model (VAT), reduces the annotation need
by 50%. Our method also performs the best on the VideoAttentionTarget dataset.

通过引入两个新的先验条件，我们提出了首个半监督的凝视追踪方法。我们利用大型预训练的视觉问答模型计算 Grad-CAM 热图，并通过修正扩散模型中的反向采样过程来改进热图。我们的方法在 GazeFollow 图像数据集上胜过简单的伪标注生成基线，并将注释需求降低了 50%。在 VideoAttentionTarget 数据集上，我们的方法也取得了最好的表现。