Gaze following and social gaze prediction are fundamental tasks providing
insights into human communication behaviors, intent, and social interactions.
Most previous approaches addressed these tasks separately, either by designing
highly specialized social gaze models that do not generalize to other social
gaze tasks or by considering social gaze inference as an ad-hoc post-processing
of the gaze following task. Furthermore, the vast majority of gaze following
approaches have proposed static models that can handle only one person at a
time, therefore failing to take advantage of social interactions and temporal
dynamics. In this paper, we address these limitations and introduce a novel
framework to jointly predict the gaze target and social gaze label for all
people in the scene. The framework comprises of: (i) a temporal,
transformer-based architecture that, in addition to image tokens, handles
person-specific tokens capturing the gaze information related to each
individual; (ii) a new dataset, VSGaze, that unifies annotation types across
multiple gaze following and social gaze datasets. We show that our model
trained on VSGaze can address all tasks jointly, and achieves state-of-the-art
results for multi-person gaze following and social gaze prediction.

本文介绍了一种新的框架，用于联合预测场景中所有人的凝视目标和社交凝视标签，通过使用包含图像特征和个体的凝视信息的时间变换器，以及一个新的数据集 VSGaze，该模型在 VSGaze 上的训练取得了多人凝视跟踪和社交凝视预测的最新成果。