Contextual cues related to a person's pose and interactions with objects and
other people in the scene can provide valuable information for gaze following.
While existing methods have focused on dedicated cue extraction methods, in
this work we investigate the zero-shot capabilities of Vision-Language Models
(VLMs) for extracting a wide array of contextual cues to improve gaze following
performance. We first evaluate various VLMs, prompting strategies, and
in-context learning (ICL) techniques for zero-shot cue recognition performance.
We then use these insights to extract contextual cues for gaze following, and
investigate their impact when incorporated into a state of the art model for
the task. Our analysis indicates that BLIP-2 is the overall top performing VLM
and that ICL can improve performance. We also observe that VLMs are sensitive
to the choice of the text prompt although ensembling over multiple text prompts
can provide more robust performance. Additionally, we discover that using the
entire image along with an ellipse drawn around the target person is the most
effective strategy for visual prompting. For gaze following, incorporating the
extracted cues results in better generalization performance, especially when
considering a larger set of cues, highlighting the potential of this approach.

通过调查视觉语言模型（VLMs）的零样本能力，探索提取各种上下文线索以提高凝视跟踪性能的可能性。